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ABSTRACT 

This study was the second year of a project that 
attempted to formulate criteria for project evaluation and funding 
decisions. Activities were designed to produce measures of project 
impact for vocational education projects. The second year's work 
focused o^ refinement of impact varial}les, /and a survey of directorjl 
of vocational education concerning their priorities for impact 
variables. Local education agencies (LEAs) | in ^^lew York State were 
asked their opinions about whether the impact variables were valid 
and whether it would be feasible for them jto. collect the data 
required. Six project impact variables were considered the most 
valid. (1) Students trained have positive I at-tituldes toward work. (2) 
rTraluiuy increases student employment options* (13) Employers are 
satisfied with graduates of the progra:m. ju,) Graduates are working in 
occupations for which they are trained^ (5)' Pro j;ect objectives ^are 
^Ifulfilled. (6) Training .objectives are met^ in the most cost-effective 
^manner. The lEAs queried indicated that cofllection of necessary data . 
would be feasible but that they could use /support with data 
collection in several _areas. The next stepA required are a review of 
the impact rating forms and development of additional related rating 
forms and the preparation of a funding and e.valuatipn. handbook for 
LEAs. (Author/JH) ' \ 
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. ABSTRACT 

During 1977-19/8 the Department' of Occupational Education Super- 
vision funded the second year of a study to develop definitions of 
project "Impact" and to relate these definitions to funding decisions 
made by DOES. The definitions and priorities for high impact projects 
in' vocational education are important for both rfunding, that is the 
al locati.ons of/ money to individual projects, and 'to evaluat ion--pro- 
viding guidance to local education agencies in determining important 
outcomes for evaluation. Focusing on the same set of variables in 
funding decision making and in evaluation will assis": both state de- 
cision makers and local directors of vocational education to have 
similar priorities. 

During the first year of the project, the resea-ch related to 
decision making and priorities was reviewed, an annotated bibliography 
was compiled, and a pilot study was conducted with DOES staff to test 
out the method of examiming priorities. The current year's work focused 
on further refinement of the "impact" variables to relate them to the 
1976 Amendments (PL ^^k-k^l) and a survey of Directors of VocationaV 
Education to determine their priorities fjr impact variables, as wel 1 , 
as DOES staff priorities. Interviews with several directors assisted 
IRDOE staff to refine draft quest ionna i res des i gned : 1) to obtain priorit 
for impact variables; and 2) to obtain sample distribution data for the 
predictive and outcome impact variables. Also, LEA opinions on the avail 
ability of data were collected. Responses to the questionnaires provide 
one in"dication of validity (the relationship between DOES and LEA prior- 
ities) and feasibility (extent to which data are current ly ava i labl e or a 
be made available). 



FOREWORD 

As a downstate outreach of the New York Star.e Education 
Department's Research Coordinating Unit, the Inst.itiJte* for Research 
and Development in Occupational Education (IRDOE) has attempted to 
identify problems or areas of concern in occupational education and 
configure ways and means of ameliorating or eradicating them. One 
such area of concern was the application of "risk money" to proposal 
submitted to the Office of Occupational and Continuing Education for 
funding. Another, parallel, concern was the basis upon which proj- 
ects, once funded and completed, were evaluated. 

In both cases, although personnel engaged in the tasks were 

consc lent iousl y appl y ing their understandings of the terms to the 

' t! . ■ ' ' • 

processes, it seemed that there were considerable variances in 

definition of terms and criteria: Accordi ngly , i n FY '75, IRDOE 
undertook an in-house review of 1 i teratureXand it^formal assessment 
of practices and, when satisfied that a need. existed to standardize 
practices, developed a proposal in FY '76 for a study in depth- 

This document reflects a two-year effort designed to, first, 
develop definitions and criteria applicable to funding decisions 
and, secondly, to test parallel criteria In^ the assessment of proj- 
ects funded and completed. That the repor^ is timely is evidenced 
by the DHEW s Off ice of Education (OE) publ i sh ing, at this writing, 
RFP 78-67, "Development of Criteria to Measure. the Effectiveness 
and I mpl ementat ion of Demonstrations in Vocational Education," 
It is hoped that the materials herein will- contribute significantly 
to the OE thrust as well as. that of the I;::w York State Education 
Department. 

Le^ Cohen, Ph.D. / 
Director, IRDOE 
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1 . Introduct ion 

Overview . > 

This study of the validity and feas ibi 1 i ty of , impact ratings is 
the second year_of_a. project concerned wi th 1 inking evaluat ion and fund- 
ing decision makinc^;;^: During the first year of the project, the research 
related to funding decision making was reviewed and the process was iden- 

ti,fied by which evaluation data -and data requested for funding decisions 

i' . • - ■ - 

could be linked. The' pilot work in the first year suggested that raters 

i . 

could reach consensus on the major outcomes ^that could be included in 
the evaluation of vocational education programs an J also consensus on the 
• ••predictive'' impact variabl-is for which information could be prov^ided 
at the time of funding. The consensus expressed by those conce/ned with 
decision making in funding ^general grants under Subpart 2 of the VEA 

V . . . 

'legislation suggectid that the second year of the project should be de- ' 
voted to developing a fi-hal set of both predictive and outcome variables 
for funding and evaluation resp^jctively; and to examining the 'consensus 
among fundi ng "dec i s ion makers and funding applicants as one estimate of 



the "valid-ity" of the major variables to be used in funding decision ^ 
making and evaluation. 1 

\ The second year of the project, then, has carrTed oyt a Series of 
activities designed to result in a final list of variables with priority 
in evatuatioji and funding decision making. The activities included the 
development of .a series of questionnaires designed to assess the prior- 
ities of two fna^on^^^^^^^^ and decision making- 
the Department of Occ^ Staff and the Local 
Education Authority (LEA) vocational education staff who receive funding 
under; the |a|ic grants sect ion. of the VEA of 1976,' One questionnaire was 



designed which would permit each -group to express their rank order of 
importance ahd rating for a set of variables which could be known at j 
the time- of funding (predictive impact variables). A second set 
.(outcome impact Vjariables) could be included in evaluation of VEA grant 
programs. The rank orders and ratings permitted the development of two 
sets of data. The first set v.'as concerned wi th the consensus among LEA 

raters in different regions . and>\arge cities of the state, and the 

I , * . . . ■ ^ 

agreement of the LEA staff overaH with the DOEb staff. The. second set 

of data is the weights or the degree of importance attached to the 

vari^rbles recei ving the h i ghest . pr ior i ty ^mbng the sets of^ raters sur- ' 

veyed in the study. 

In addition to cletermin ind the priority and criterion weights for 

major predictive and outcome variables, two additional questionnaires 

were designed to ^provide estimates, of the categories on which projects 

might be described; for each of .the impact scales* That is, for a variable. 

determined tO;^be a/fnajor priority and given a high weighting, such as 

number of local Jobsv.avai lable In the arej of the training program, some 

Index is need ed to. relat of employment to potential 



number of graduates of ,^f1ie •program.r One way of providing a "score" for a 
.project pn. the pryprl^^^^^^^ is to categorize each project into one 

of tht^e categories^^^^^ average and weak, on each impact variable. 

In order to categorize projects, some idea of the values for each of the 
categories need sr*to be avail/ible through data or subjective estimates 
(or a cof7ibinaitjon':p:f^^ and DOES/LEA staff estimates.) 

The secpnd . twQ^^ des i gned /or the study were then aimed 

at ^ol 1.eqt|.ng:d^^ related to the need to provide rater categories. These 
two questionnaires had several purposes. They examined' the amount of data 



:,re^^dily ava i lat a±(LEAs, and they ai*l so provided for LEA staff to 
suggest--t'ha*t--ffie data would be available if given a.dvanced notice or 
would bfe very difficult to collect. The -quest ionna res served the 
further purpose of. beginning a dialogue between DOES and LEA staffs 
on the importance of providing data for funding deci sion making and 
evaluation of VEA-funded programs. 

In-summary, the first^year of the project examined the methods 
currently available to DOES staff in establishing priorities for VEA 
outcomes and the rjlationshiii^ between specifying important variables 
for funding decisions as ,they\ related to evaluation of projects. The ' 

second yei^r has determined one\aspect of the validity of the major 

/ \ ' , ■ ' ^ . 

variables for /funding and evaluation, that is the relationship between 



the prioritie.s esr.ablished by LEA and'DOES staff, and secondly, has 

./ . ■ ' V ' ■ * ■ • . 

provided some estimate of the -d I st r i but ion (d i f ferent va 1 ues projects 

may have on ^the impact var.iables)\wi th a view toward establishing rating 

^ ' I . \ • ■ , j 

categories for funding decision making ?nd outcome eva 1 Jat i on The 

procedure ; s and results of the! study .are' presented in Chapters 3, an d 

Chapter 5 provides an overa.p-summar^ and recommendations for next'.st^ps. 

* ' ■ ' • . . \ ■ ; ' y-:-. . .,/ ■. 

The second part of this Introc-uction describes a recent y-conducted study 
for community colleges which Is similar to the work being de velo ped at the 
high* school level 6y,D0ES. V 
Recent Studjes 



The review of the research 11 terature\^ pre'tented in the final report 
of the first year of the project, An ExploAatory Study .of the Impact of 
Vocat ional Education: Impl icat Ions for Fund l\ng , described two areas of . : 
research related to the "current project . The f i rst area is the early 



V. )rk in decision making which/proy'ides several ways to estimate, the 
utilities or values attached to particular varfables or outcome? of 
decisions and the second was a beginning area of rese^irch in linking . 
studies of decis ion making to funding decision making, "^wo stiidie^^ 
have appeared in the past year which merit description in terms of 

thjsir/relat ionship to the current project The first is a review and 

1 ' ' ■ x * . ♦ 

-1 /" ' \ ' * 

set of studies by Esnhorn and Hogartfrx (1 978 , in press) which examines 

the. confidence, thgt judges have in thel r deci sion making. They examined 

the contradictio|i between demonstrated evidencie on the fallibil,ity of 

human judgment ancl. ths **seemingly unshakeable confidence people exhibit 



in their judgmental <ibi1ityj' They str^ss^d three key variables in the 



judgment process. 7he first is the str;icture of the judgment task, the 
second is the extent to which people can observe the outcomes of their 
judgments. As they note, most studies have simjply'correlated judgments 
with criteria. How individuals or Judges then decide the action to take 
has been neglected. As they point out*, in real worl d s I tuat ions , judgments 
are made for the purpose of choosing between actions., Thi s .means that 



outcome Information, such as, evaluation da^a, v/hlch-i^ aval 1 able only 
after the actions' or funding decisions are h^de. Is frequently the only 



source of feedback with which to compare judgment. .Their thesis is that 
in order to better understand judgmental ability, ift is necessary to con- 
"sider the judgrfient, the act ions taken , and the outcome feedback together. . 
,Their*argument provides a further rat Ipnale for the current projects efforts 
'to link the outcome and predictive I mpact'^v* statements so that. evaluat^Ion • 
data, can "feed-back^' to mod i fy future judgments. 



Einhorn and Hogarth also point to a research problem in strictly 

•correlating judgmentB of the impact or success of projects anvd the judges' 
- . /. 

Iry? t?al jrat ing of projects. Research problems are that one can only 

look at Accepted projecvts to see 5f they have been successful, s'^jce un- 

\ . • ' . ' ■ ^ ' . ■ 

successfililappl icants are not funded. The ideal validity study for ^ * 

j/idgments would include_f unding both projects judged to be of high impact 
land a smaller subset of\ those judged to be of low potential, and then to 
observe the success rate. However, for practical and ethical reasons this 
type of study is never carried out. The result of this research design 
problem is that the "treatment" of receiving a gi^ant is completely corn- 
founded with judgmental accuracy.. That is, 'giving a LEA a grant may give 
them time and resources to do more and better work as opposed^to applicants 
who are not funded, . 

The limitations of correlating judges* ratings of high JmpacJt projects 
with criterion or independent judges ra^ ings of high impact projects has 
implications for the present project. The current project lias, therefore, 
taken a somewhat different first approach to validation, and that is to 
examine the relationship between LEA staff rankings and .weight ings of the 
priority.of impact variables with those of DOES staff. 

The second study which is related to the work of the current project 
was conducted by Ory, Harris, and Clark (1978) and also reported in a 
second paper by Ory, Ha rr i s , Due i tt , andClark (1978): This sturdy devel- 
oped and field tested a vocational education evaluation model for programs 
at the community cc liege level. The metropolitan community colleges of 
Kansas City, Missoiri were involved in the study. An advisory group of - 
163 state and local educators, legislators, college trustees, and business- 
men were asked to rate six criteria for vocational education program eval- 



^nation. These six cri tenia .were: 

1. The program's rela^tionship to the job mari- it profile 

2. The program's success in meeting vocational aspirations of 
• cl ientele 

3- The program's success in terms of student support 

k. The program* s. level of community support 

" 5* The program's cost-effectiveness 
/ ■ / • \ 
6. The program's success in reaching the handicapped 

and disadvantaged 

These six criterion statements were rated, each statement pafred against . 
every other statement (in the paired comparison procedure), and 100 
usable responses were available from the survey. These responses were 
used to develop the crit'eriipn weights/ The fhree statements weighted 
most highly were: the prpgrams's relationship to the job market ' prof i le; 

the program* s success In meeting vocational aspirations of clientele; 

• ■ ..■ • '■ . . ■ ■ 

and the" program' s success in terms of student performance- 

The weights are used to emphasize . the important criteria for eval- 
uating programs. A series of assessment instruments were developed so 
that the vocational programs were assigned a quality rating of strong , 
adequate , or weak (3, 2, or 1 points) on/ each of the six criterion meas- 
ures. The criterion qual i ty points are/ then mul tipl ied by thei r respect- 
ive weights and summed to provide a program's aggregated criterion score- 
The comparisons based on aggregated criterion scores as indicators of 
^program^qtial i ty are intended to be useful in the decisions of maintaining, 
modifying, or terminating a vocational program. 

Of particular interest to the current project was the definition 
of sub'-cr i ter ia within each of the, major criterion areas. Figure 1 shows 



7 ■ ■ ' . 
Figure 1. Evaluation Score Sheet 
(Ory, Harris, Dueitt, and Clark, 1978, p. 10) 



^. Attainment 
^ . Ratings 

J 'J ^ . 

^ ^ Jl^ 

^ J J? ^ 



^ S <o ^- "^'^ '.Criterion I. LocaV Job Varhet Needs 

^ (circle) i , 

A 32 X 3 @ 1 0 » The occupational conrmunity's expressed r.t-r-d for nr<>duates 

B 29 X © ^ 1 0 £i Pvo^rm craduates V need for eDplo^Tient {?b findina ,iobs) 

C 28 X 3 © 1 0 - ^ The occupational .co.iMunity's expressed nrcd for the t^/pe of 

^ education offered by the prorrar. 

D 11 X 3 ©1 0; « 4^ Projected employment needs reported by tlie K.C. Mc-noo'/^r P.enort 



Total J£f X .30 « 6? 7 Weighted Criterion 1 Score 





Criterion Il\ Past and Present Student .Needs 

■ M Enployed graduate satisfaction with the procran's job preparation 

■ ^Employed oraduate satisfaction v/ith the f-no'-zledge rffered 
» ,2V Employed graduate satisfaction with the sKills trainino offered 
= ^Emplcved Graduate ratinos of the oronrair.'s duality 

■ 4P Current student satisfaction with the sin'lls traitiino offered 
^ - - ■ jfy Current student satisfaction vn'th the knovledne offered 

X CD 2, 1 0 .-^Current student ratines of the program *s quality 

Total Jjif^ X .26 ^ fc3.7 Vjeiohted Criterion 2 Scorg - 

Criterion III. Graduate Job Perfbmance 

• £4Emplc;yer satisfaction with the graduate's overall oerforrance 

* Jh Enployer ratinos of the ouality of the graduate's overall performance 
' ^ En:ployer reports of spending less, equal or nore time on entry 

training . 

■ ^ Employer reports of saving, breaking even^ or losing money on entry 
t' Aining . * * ' - 

Total ,^jaQ.X .24 - J£2_ Weighted Criterion 3 Score ^ ' 

Criterion IV. Occupational ComnunUy Support 

^ Si^ Employer (of graduates) willingness to hire another nraduate 

* 22. Occupational Community willingness to hire prooran nraduates 
" Employer (of graduates) ratings of program ouality 
" At Occupational Cormunitv ratings of program qualitv 

■ Occupational Community awareness of, the prdgram*s existence 

Total X .03 = ^ A/ Weighted Criterion 4 Score . 

Criterion V. Cost F.ffectiveness 

• lux Co^t PC*" Credit Hours 

" JL1£ Cost per Contact Hours ^ 
Total .^DD X .07 = JJX^-e^Qhted Criterion 5 Score x ' . 




Criterion VI. Success in Heaching Handicapped ^ Hisadvantaqed 

A 41 X d) 2 1 0 « Handicapped and disadvantaoed student satisfaction with proqram 

B 28 X 3 2 ® 0 • ^ Availability of support services (i^e.. tutoring, financial aid) 

C 19 X 3 (2? 1 0 " ^ Facility construction and/or alternation to meet special needs 

D /12 X 3 2 CD 0 « Jja, Evidence of program's recruitment efforts • 

Total JO/ X .05 « lO-f Weighted Criterion 6 Score 

ComposUe Score » A}!, U (Sum of all six weighted criterion scores) 



r ; 

Figure!. Evaluation Score Sheet / — 



an example of an evaluation score sheet {Ory, Harris, Dueitt, and Clark, 
1978). Each of the major criteria has sub-criteria, sub-crlteria weights,^/ 
and a rating of strong , adequate , weak, or no data . For example, criterion 
1 on local job market needs has/four sub-criteria: the occupational com- 
unity's expressed need for graduates; the program graduates' need for 
employment {% fmdSng jobs); the occupational community's expressed need 
for the type of education offered by the program; and projected employ- 
.ment-^needs reported by the Kan sa s'^C i ty. Manpower Report , These data were 
collected from employer and student surveys in order to provide individual 
program ratings of strong , adequate or weak . ' ^ 

The project establ ished the rating categories in conjunction with 
administrative decision makers. ' For example, in order for a program to 
be rated STRONG on sub-criterion 1, "program graduates' success in finding 
employment," Ory et al report that at least $0% of the graduates ^needed 
to be .employed in training-related fields. At least 75% needed to be 
employed for a rating of ADEQUATE, and below 75% programs >received a rating 
of WEAK. The proficiency levels of 30%^ 75%^ and less than 75% were used 

■ • '■ ' ' " ■ ^ i 

to assign the attainment ratings of STRONG, ADEQUATE, OR WEAK, respectively, 

• ■ - ■ • / 

to that particular sub-criterion. The unique set of . prof iciency levels: 

'' ' \ •' ^ I 

were based, as mentioned earlier, on. the standards of quality desired by 

' i 

the Individual college's administration. \ 

In the pres.ent project, in contrast, a first step has been to c^d- 
struct questionnaires to go to LEAs to collect information on the dis- 
tribution of major variables. One difference.between the present s/tudy 
and the Ory et al study, is that the latter collected data on completed 
programs. One of' the major problems, as reported later, is that the 



funding decisions made by DOES are not always directly program-1 inked, : 
For example, some grants may cover equipment for several programs. The 
"entity" being funded is not as well defined as the program being eval- 
uated in the..<romni^^ study. 

\n/^r(ie^^ to evaluate ten vocational programs of 

tlje Kanslas City metropolitan community colleges, over 16,000 question- 
nai res^^rffe^ sent to varj^^ data sources, including current students, 
graduates, ^^t>pbii^ business persons, employers, and program 

chairpersons. Among the . quest ionna ires were .evaluat i on checklists for 
facilities (to determine accessab i 1 i ty for the physical ly handicapped), 
a program chairperson's checklist for level of program participation 
offered to hai^dicapped and disadvantaged st »dent5, and a cost-evaluation 
form used by evaluation personnel. 

Ory et al reported that criterion scores across allten programs 
range from 0 to 300, while composite (weighted) scores had a range from 
92 to 258. Of the ten programs used for field testing, three averaged 
criterion ratings of STRONG, six averaged ADEQUATE ratings and required 
modifications (in areas suggested by the program cr i terion ratings), and 
one program was rated WEAK. According to the authors, these ratings 
supported subjective judgments by administrative personnel ^pr ior to the 
evaluation and informal data collected with the questionnaire. 

!n summary, the two recent studies described here suggest the im- 
portance of linking outcome data to data that can J?e known at the time 
of funding, and further suggest the need for providing both evaluation 
and funding decision makers with categories that can be quantitatively 
defined, either on the. basis of judgments by local administrators and 
evaluators or by sample data reported i-n the pr,esent project. 
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II. Development.of Questionnaires 197 7^1978 
Rating procedures in use at SEP 1977*78 ;\ : 

In line w.Ith the new regulations for the VEA of 1976; project 
proposals for fiscai year 1979 he/e followed a different series of 



:eraures in 



\ 




procedures in the department. Each prqject application is first 
examined on a series of screening criteria. The initial screening 
criteria are in two breas. The. first is initial project screening . 
crUeria, comprised of six items which are checked yes or. no. The 
six items are as follows^. ^ 
1. The agency was involved, in regional planning for 
\ occupational education. ' 

/zN^ There is written documentation of local advisory council 
involvement in developing the proposal. " . 
3- The agenQy has maintained effort- (aggregate or per capita 
expenditures) from the previous y-ear. 

The proposed activities are allowable under the legislation 
(use statement::,of allowable activities), 

5. The proposed expenditures are . al 1 owabl e under tKe legislation 
(use sta-tement of allowable expenditures). 

6. The' agency has a ppl icy on sex bias, ^stereotyping and dis- 
cr imi^nat iorf o.n fife ^nd proposed activities are consistent 
with the poi icy. ^ ' 

y The second s,et of screening criteria are on economic and manpower 
needs documentation. Thehe are four statements in the economic/manpower 
needs screenjng criteria and they are rated on a scale going from high, 
medium h i gh^.jiied i um low., to low, and receiving respectively, 4,3,2, or 1 
points. The four screening criteria statements In this area are: 



1. The project serves an, area of high economic .need as . 
evidenced by hi^h rates of unemployment, concentrations 
of low income fami lies and other economic indicators. 

2. Information^on labor market needs deals specifically with 
needs of theVegion or area served^by— the educational agency 
(regional data preferable f6r schools serving regional 
labor market: 'Statewide data acceptable in cases where ^\ 
agency has area-wid/a. or statewide area of placement). 

3» Information on labor market heeds is based on; objective 
■ data ( not 1 imi ted to opinions, personal knov^ledge or testi- 
monials) and is corroborated by at least, two data sources. 



Information on labor market needs i^s directly related to the' 



specific program (instructional or support serv ices) seeki 
funding under \^EA. ' 



ng 



The projects needy to receive a score of 8 or greater to continue 

in the review process/ Figures 1, 3, and A show the general flow charts 

. • ' - '* 

for the review and approva 1 deci s ion ipaking process, for each of the 

■I 

three main population categories under the VEAl egi slation - general 
population, handicapped and disadvantaged, and adul t population 
As shown in the flow charts, the present procedure goes from the 
screening statements, including regulatory issutes and economic/manpower 
needs, to the ratings of instructional prograqi qliality and program pianage- 
ment quality. Program management ,qual i ty -and instructional program Iqual i-ty 
scales are shown in Appendix The management criteria include 

ratings from 1 low' to k high in the area of statement of Reeds, objectives, 
activities, costs, continuation, evaluation, articulation, and 
agency experience/ past performance. The instructional project quality. 



Jlow Chart of VEA Reviev? and ApDroval' Decision MakinR Process: 
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Flow Chart of VEA Rtvicu and Approval Decision Making Process: 
Handicapped ;& Disadvantaged Populi^Hlons 



Handicapped: i/. Disadvantaged 
Rroiects 



./ 



t 
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level job ysklUs : or give studftits 
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/ 
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If job. skills. 



/ 



1 
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to be foltowed 



Use flow chart for 
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iPast Perf orvance i 
of Progran I 

I ■ 



"j Quality of 
i Continuation 
Program 




Medium High 
4ch 1st 







Improving Existing j 
Programs j 






i Qiifalitr of Program ; 


Quality 


Ranking 




^tedium High 
5th 2nd 



! New Programs i 



: Quality of 
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Flow Chart- of VE.V Revievr. and. Approval Decision Ilaking Process: 
; • \ Adult •Populations - 
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criteria Include. ratings on heeds, objecti ves activities, staffing, 
equipment, facilities, costs,. and agency experience/past performance. 

The state rating scales for program management and instructional 
quality develop sub-categories for each of these areas and provide 
four-point rating scales. These procedures may be contrasted with 

those reported by Ory c t a 1 , where there was a joint effort by adminis-- 

■ 

trators and evaluators to set category defi ni t ions , ^ahd/wi th the effort 
in the present project to^ provi de somie est imates of values for the 
categories based on project questionnaires. The next sect ion descr ibes • 
the rat ionale and- the quest ionna I res . for the present project.. 
Rat ionale ^and. major sect ions of study ques,tionnai res .^; 

Following the outcomes of the 1976-1977 study, a summary of outcome ' 
and predictive Impact statements were developed for discussion with. the 
staff of DOES. These summaries are presented in'Tables 1 and 2. IRDOE 
and DOES' staff met on January 16, 1978, to review*the previous year's work 
and to develop the revi sed" i mpact statements based -On the I976, VEA amend- 
ments. At the meeting It was decided that LEAs shoul d- al so rank order the 

impact . statements . Because gf thjs. decision, some' of the impact s^tate- 
ments that previously had low rank§ were left In the set. ^ 

On .l^.he basis" of tKe^d iTcuss ion at- the;..mee^^^ three questionnaires 
were developed, (luest ionna i re 1 provided for ranking and rating of 
predictive and outcome impact statements, (iuest ionnai re 2 provided for 
the collection of data related to each predictive impact statement , to 
ascerfa.In the data whidh are readily ava i 1 ab 1 e ^to LEAs , data that could 
be col lected given advance notice, and data that LEAs considered Im- 
possible to collect. Similarjy, (iuest ionna I re 3 provided' for the collecti 



Statenehta that received the highest ratings and rahKihj. * : 

(A) /Training objectives are met with minlinal cost 78r 
'Student. ' ;' ''\ ' ■' ' 

. .^Ci_-^J?8??»» graduates arc working In occupatioiis for which 
t?hey were trained. . _ 

(E) Vocational education needs of special groups are met. 

(G) Training Increases student employment options. 

(J) Employers are satisfied with graduates of program. 

(K) Job satisfaction of students trained is iiicreased. 

(L) Students trained have a more positive attitude toward work, 

ratings and rankings, but should be 
reconsidered (Parallel forms were ranked high amoig predictive statements) 

(F) Students are trained for occupations traditionally dominated 
by the opposite sex. 

(H) Students learn career planning. 

(D) Program can be replicated in other settings. 
Statements that received low ratlrifes and might be deleted: 
/(B) Large numbers of students are trained. 

(I) Students trained continue their e4)yicatlon. 



/ 
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Table 2. PREDICTIVE IMPACT STATEMENTS - January 1978 

Statements that received the highest ratings and rankings : ; 

(A) Training objectives will be met with minimal cost per 
student. 

(C) Students will be trained for occupations where jobs nre 
available. _1 . • 



(D) Program will be replicable in other settings. 
(E> I Vocational needs of special groupsr^wi 1 be met. 

(F) Students will be trained for occupations traditionally 
dominated by the opposite s*ex. . ^- 

(G) Training will be provided to iTicrease students* employment 
options. 

(H) Students will learn career planning > 

New statements that 'should be .considered (Parallel statements x/ere 
ranked high among outcome statements): 

1 CJ) Students will be prepared to meet job ^requirements as , 
specified by prospective employers, (e.gi , employe rV 
ratings of program performance objectives in terins pf job 
requirements) . 

(K) Training will increase students' job satisfaction, ^e.g., 
prospective students will rate course goals/curribuluici in' 
relation to being able to do their work well) 

Statements that received low ratings and ranking and might be deleted; 

(B) Large numbers of stii^dents will be trained. 

(I) iNew program will serve students ' iriterestV Ceven thoug^^^ 
!ar€i not available Ibcaiiy) 



;,of data for each of the outcome impact statements. The procedures 
for developing these questionnaires were to identify the diaM '^feAu i red . 
for each Impact statement, to develop an index or single' number ais^a 
basis for grouping or ^categorizing projects bn a rating scale on that 
statement, and to then reduce the amount of data actually asked for in 
the questionnaire to the bare minimum. This letter undertaking was 
assisted in large measure by the cooperation generously given by three 
LEAs. ' Draft questionnaires were reviewed by Mr. Thomas Castelli, by 
flh-. Donna Santa and her staff , and by -Dr, Howard Friedman and his staff . 
On the basis of their comments IRDOE staf f s ijnpl i f led the questionnaires 
still further for the survey. 

The three questionnaires used in the study are given J n Appendices 
B, G, and D. The surve> procedures are^desxsribed below. 



f ', : . ; 



I I I > Sample Survey 

Procedures 

IRDOE staff Identified one completed project from FY 1976^ and one 
project appHcafion for FY 1979, for each BOCES to serve as the basis for 
completing Questionnaires 2 and 3 In the survey. For the majority of 
BOCES and large ci ties, there was no sample project selection, since 
there were few projects l isted and the selection criterion was to 
draw on a general .population, secondary level project as the first 
preference. In a few cases, adult or special population projects were 
used. / 

A separate procedure was followed for New York City .for FY 1976 and 
^ FY 1979 project selection. In cooperation with Dr. Howard Friedman, of 
the Board of Education Center for Career and Occupational Education, in- 
dividual projects were selected for New . York City.^ Ten. were selected 
for the predictive and ten for the outcome questionnaires. The basis for 
selection was to include a range in terms of the dollar amount allocated 
to the grant?* and a range in terms of the type of occupation covered 
by the grant. ^ 

During April and early May, questionnaires were mailed to LEA and 
large city directors of vocational education or, for New York City, to 
individual project directors. The majority of questionnaires were sent 
out with a letter requesting cooperation from Dale MV'Post, Director of 
the Division of Occupational Education Supervision. The New York City. 



'•Umbrella" grants including a number of individual project are often 
used for New York City. The most comparable data would be for In- 
dividual projects with the "umbrella" grant, so these were identified 
for the survey. 
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questionnaires went out under the cover letter by George R. Quarles, 
Chief Administrator for the Board of Education of the Center for Career 
and Occupational Education. Following the. Di rectors ' letters; a letter 
from the Institute for Research and Development in Occupational Education 
requested cooperation also, explained the purpose of the survey in more 
detail, and identified the.VEA numbers and project titles on the question- 
naires. Also, the review process was described asking respondents to 
include data where they were., ava i lable/^but where data were not available 
to mark each question as to whether: the data could be supplied if given 
advanced notice; the data were not accessible without considerable 
effort; or the data were impossible to collect. 

Returns: ■ 

LEAs were requested to call if there were questions about the 
survey and to return the surveys within approximately three weeks. Ques- 
tionnaires were returned during May and June. Two telephone follow-up 
calls were made for non-respondents and the final number of questionnaires 
returned were as follows: ^5 BOCES questionnaires were mailed and 39 . 
returned, for a response rate of 87%. For the large cities, 63 question-' 
naires were ma i 1 ed out and 42 questionnaires were returned, for a response 
of 67%. ^ All three questionnaires were sent to each BOCES. 

All of the, cities' programs ' recel ved a Questionnaire 1 and either a 
Questionnaire 2 or 3 depending on whether that program'was funded in 
FY 76 or FY 79. Out of. the 29 Ques't.ionna i re 2's that were sent out, 20 
- (695) were returned and '22 (65,?). out of 34 Questionnaire 3's were returned. 
In cases where It was obvious that the same person filled out more than one 
copy of Questionnaire 1, only one copy (selected at random) was used in the 
data analysis', therefore, altiftough; 42; Que Ps were returned^ 

only 33 unique copies were used -for .the analysis reported in. Part 111. 



- IV. Results 

A. Ratings and Rankings of Impact Statements (Questionnaire 1) 

Three groups responded to Questionnaire 1, providing rankings and 
ratings of importance of predictive and outcome impact statements: The 
first group is comprised of di rectors or project directors in BOCES 
(n=38), the second groCjp consists of respondents from the five large 
cities in the state (n=33) , .and the third group includes individuals who 

are supervisors in DOES (n=5) . The data from the rank order ing. of the 

. ... • 

Importance of the predictive and outcome statements are given in Tables 

3 and 5. The tables show the overal 1 rank based on the mean ranks 
given each statement by each of the three groups and the total 
group. Tables k and 6 provide the means and stand.ard deviations for the 
rating of importance for each statement. The ratings were originally given 
on a scale from one (low) to twenty (high), but were transformed to a mean 
of 50 and a standard deviation of 10. The transformation was used to 
place each rater's set of ratings on the same scale, so that comparable 
numbers could be used for summary purposes. 

The amount of agreement between raters was est imated. for the DOES 
staff ratings. Kendal l' s. Coefficient of Concordance (W) was computed for 
the ranks given by DOES staff tb the predictive and outcome impact state- 
ments. W had a value of .638 for the Predictive impact- statement ranks 
(X2 = 28.7 p<^.00l) and W = ,689 for the Outcome impact statements. 
(X = 30.9 p >\001). These values show a highly significant amount of agree 

ment among the set of DOES staff judges. (W can take a value from 0 to 

^ ■ 

+ 1, and is defined as the ratio of the obtained variance to the maximum 
variance possible.) 



Table 3. Overall Ranking. of Predictive Impact Statem-^^nts 
by BOCES, Large City, and DOES Staff 



Predictive Impact' Statements. Ranking 

Large 

. BOCES ^ Cities^ DOES^ Overall 



occupations where job's are 
avai lable 


h 




1 
1 


I '. 

— 3 


2. Project objectives are stated 
In measurable terms 


5 


! H 

5 


3 


5 


3 . Nn ^py H T Q/r r ! m I t* I on ui i 1 1 

made in Yecruiting and placing 
students in vocational programs 


7 


/ 


7 


7 


4. Training objectives will be met 
in the ntost cost effective 
manner " 




6 


k 


6 


^ Lc^rap niiiTiK^r of ct'iiHonf'e uii 1 1 

^» L.ai vjc iiuiiiuci Ul aLUUcnLS WI 1 I 

be ;tra Ined 


TO 
1 u 


Q 

9 


10. 


9 


O. Trainina will hp nrnuTHoH i-o 

w» iiaiiiiiiv^ will U KZ piUVIUcU .LU 

increase students' employ- 
ment options 


1 
1 


3 




,2 


Sf'lJnPnt'Q Ui 1 1 1 Ko nr"onai*ftH 

/ • .«^uuuciiu3 WI 1 1 uc.prcpareu to 
meet entry level.^.s^k? 11 require- 
ments as spec! f leck by prospect- 
jve employees, (e .grVmployer ' s 
ratings of program performance 
objectives In terms of job re- 
quirements). 


Z ■ 


■ I 

1 
1 

■ \ 


2 ■ 


1 


8. Program wi 1 1 serve students' 
Interests. 


3 


k - 


5.5 




9. Program Is articulated with 
local post secondary 
InstI ttitlons 


8 


8 




8 


10. Program will be replicable in 
other LEAs 


-9 


10 


: 9 ■ 


. 10 



^Overall rank based on mean ranks^ See Appendix F. 



I ; ' ■ ' W = ., .-896 

; : * X^W (9 df) = 2i».l8 .01<p<.001 
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Table ^. Ratings of Predictive Impact Statements by BOCES, 
Large City, and DOES Staffs: Mean and Standard 
Deviation of Standardized (T) Scores. 



Predi ct i ve 
Statements 



Impact 



BOCES 
(N'38) 
Mean S.D. 



Cities 



Supervisors 



All 
(N =76) 
Mean S.D. 



1. Students wll be 
trained for occu- 
pations where jobs 
are aval lable. 


53.35 7.12 


56.93 7.10 


60.03 3.31 


55.34 7.19 


^2. Project / objeqtlves 
are stal||d in meas- 
urable terms. 


51.36 7.77 


51.87 8.85 


53.82 2.66 


.51.74 8.00 


^ 3 • No . s ex discrlmlna" 
t ion wi 1 1 be made 
In recruiting and 
placing studeftts 
In vocational 
prograpis 


'♦0.31 0.4/ 


H7.09 7.27 


f" 1 !• 0 ft* Tft 

51. HO 9.79 


li T ft ft T ft !• 

^7.99 7. Oh 

* 


Training object- 
ives will be met 
in the. niost cost 
effective manner 


52.75 5.92 


48.22 6.17 


51.91 8.98 


50.72 6.54 


/ 

/ 

5. Large/ numbers 
-of students 
will be trained 


39.35 .6.03 


ko.yk 7.56 

\ 


36.95 3.27 


39.80 6.63 


6. Training will be 
y/provided to In- 
/ crease students' 
/ employment options 


59.80 5.32 


53.83 7.03 


53.01 -5.32 


56.76 6.78 


/^7- Students wi 1 1 be 
' prepared to meet 
entry level ski 1 1 
requirements as 
specified by pros- 
pective employers 


56.90 5.33 


57.26 7.^2 


59.01 S.84 


57.19 6:47 


8. Program will serve 
students ' interests 


55.35 6.83 


53.38 7.49 


54.12 3.43- 


54.^1 "6.97 


9. Program is articu- 
lated with local 
post secondary 
institutions 


'•3.53 5.33 


4.3.88 5^66 


42.45 6.95 


* -- 

43.61 5.46 


10. Program ,wi 1 1 be , 
repl Icable In 
other LfiAs 


36.50 kAS 


37.28 6.11 


.37.23 7.46, 


36.89 5.38 



Table 5. Overall Rankings of Outcome Impact Statements 
by BOCES, Large City and DOES Staffs 



Outcome Impact Statements , Ranking 



Large 

BOCES^ ' Cities^ DOES^ Overall 



1 . Program graduates are 
working in occupations 
for which they were 
trained. 


" 5 


2 

t 






2. Project objectives 
are. f ul f i 1 led 


/ 6 




., 5 ■ 


5 


3. No sex discrimination. 

occurred in student 
^ selection , training, 

and. job placement 


7 


8 


7 


7 


k. Training objectives are 
met in the most cost 
eiffectrsve manner 


k 


6 


2 


6 


5' Large oumbers of students 
are trained 


3 


■ 9 


9 


9 


6. Training increases student 
employment options 


1 


3- 


k 


2 


7. Employers are satisfied 
wi^th graduates of program 


■ 3 


5 




3 


8. Students trained have 
^" positive attitudes toward 
work 


2 


■ k 


■ :'6. ■ 


1 


9. "Students trained continue 
their education ^' 


8 . 


7 


•8 


8 


lOi Program can be replicated 
in other LEAs 


10 


10 


10 


.. 1 0 



Overall rank based on mean ranks. See Appendix F. 
W = - .809 ' " 

.X.^W (9 df) =21.8i» '.01<p <.001 



•"•"••"Table 6. Ratings of Outcome Impact Statements by BOCES, 
Large City, and DOES Staffs: Mean and 
Standard Deviation of Standard ized . (T) Scores. . • 



Out come Impact 
Statements 



BOCES 
(N=38) 



Cities 
(N.-33) 



Supervisors 
(N=5) 



Al 1 
(N=76) 



1; Program graduates - 
are workih in occu- 
patrons for which 
J they are trai ned 


51.06 7.12 


Sk.Zl 8.21 


57.6i» 6.1i» 


nean a.u. 

53.15 7.79 


2. Project object- 
ives are f ul- 
filled 


52,. 02 ' 8.17 


, 56.05 '<bM 


'51.15 5.A9 


53.71 8.75 


3. No sex discrim- 
ination occurred 
: " In student select- 
Ion, travining, and 
job placement . 


^lAk 6.62 


^6.13 6.96 


.^9.80, . = 9.33 


' i»6..88 6.91 


Training object- 
ives) are met in 
the most cost 
effective manner 


52. 9^* 5.70 


kZ.lk 6.18 

<• 


57.07 1.62 


51,17 6.3i» 


5* Large numbers of 
students are 
trained 


38.95 5.97 


^0>23 7.70 


35.^^ 1.27 


39.27 6.67 


6. Training, increases 
student employment 
\. options 


58.. 77 5.79 


52.83 7.08 


57.23 3.79 


56.09 6.86 


- -r ' 

^7. Employers are satis- 
fied with graduates 
of program 


57.03 i».95. 


53.96 7.25 


57.66 3.19 


55.7^ 6.12 


8. Students trained 
• have pos 1 t 1 ve 
attitudes toward 
work 


58.16 5.86 


5^.73 6.03 


5^.23 ■ 6.31, 


. 56.i»l 6.1i» • 

, ■ V ■■ 


9. Students trained 
continue their 
education 


^^.00 6.20 


'»5.96 8.i»6 


^1.69 7.89 


'«'«.70 7.37 


10. Program can be 
replicated In 
other LEAs * 


37.13 


37.^*6 5.39 


38.10 3.60 

i 


37.35 k^k 



Simi Tarly, W was computed for the ranks given. the three sets* 
of judges: the^BOCES, Larqe. C i ty, .and DOES staffs." The degree of agree- 
ment among the overall rank ar:;der given by the thr^e groups was^high- For 
the predictive impact statement^W = ,896 (X'^ = 2V..18, -OK p^-001 ) ,V' 
and for the outcome Impact statements. W, = ,809 (X^ '«*21 ,8^, ,Q1 <p <Co01 ) • 
The high W coefficients are reflected also In the similarity In the 
rankings and. ratings for the piredlctlve Impact statements, and similar 
"agreement for the outcome Impact ''Statements, 

The six highest statements In. the predictive rankings and ratings 

were (In overall order of pirlorlty): 

Students will be prepared to meet entry level skill 
requirements as specified by prpspecjt I ve employers. 

Training will be provided to Increase students' employ- 
ment options. 

Students will .be traxne'd for occupations where jobs are/:- 
available . - ' . j. V ) ' 

Program wIlT serve students' Interests 

^ Project objectives \are stated In meastirable terms ; 

Training objectives will be met In the most cost effective 
manner ^ 

Less priority was given to the other four statements that were concerned 

with sex discrimination, the number of students trained, articulation 

with local post secondary Institutions, and program repl Icab 1 1 I ty In other 

^ LE'As. These four statements al l received ratings beJow the mean of the 

scale, when the . rati ngs were averaged across all 76 raters (I.e.., a mean 

rating of less than 50) . "~ ' 



The six outcome impact statements that recfeived fhe highest rank 
order ahd ratings were: . • ^ 

Students trained have positive attitudes toward work 
Training increases student employment options 

♦ Employers are ^sati sf ied with graduates of program 

Program graduates are working in occupations for^ 
which they were trained 

Project objectives are fulfilled . 

Training objectives are met in the most cost effective 
manner. ... 

Again, lower priorities v/ere given to the four statements that were con- 
cerned with sex discrimination, training large -numbers of students; 

students continuing their education, and program repl icabi 1 i ty in other 

■■ ■• '* ' . i ■ • 

LEAs. 

^ . The^ significant agreement both betweeri the raters in each set of 
} ' — . ' . ■ * 

: predictive and' outcome'^impact statements, ar)d the agreement between the . 
! * ■ ' , ' ■ . ■ ■ . . ■ 

priority g i ven th^statements in the ranking and rating tasks, indicates 

that the further effl^rt to develop more objective or quantative, indicators 

for the high prior Ity statements is worthwhi le/ Sj.mi larly, the extent 

of agreement between the six statements that were/highest in the predictive 

set arid the six highest in the outcome set are rncouraging-. The 

BOCES, the large cities, and the DOES staff ca^n work further to develop 

a system, for funding decisions and for evaluation that will provide data 

for decision making that should be satisfactory to-all groups concerned. 

• * ' ' ' 

The focus In these six statements is somewhat different from the 

set of six statements used by Dueitt et a1 (1977). The six statements. 

• • ■ jt^"' ■ - . ■ 
used In that study of community collj^ges described earlier, were the 

program.* s relation to the job market , meeting vocational- aspl rations, >. 



bf students , program positi-vely supported by students, program's level 
of commujrijty support, cost effectiveness, and success in reaching the 
disadvantaged and handicapped. The two sets of six st^ateTnents differ 
in the emphasis on student and community support of the program and in 
serving the special populations of disadvantagec- and/' hand! capped. The 



statements in. the present study had only one statement concerned with 



special needs--that of sex discrimination- — <ind that statement was not 
ranked highly nor given average ratings of importance above the average 

■ . ■ ■ ^ ■■■■ ■ ■ ■ ' ■■ ■ ' il ' . " - 

for the scale used. With the exceptioh of meeting the heeds of special 



groups, the six statements In the present project seem more directly 



related^ to the VFA funding regul ations , as is^ultable in view of the 
purpose of the project and its focus -on programs and projects receiving 



federal fund ing. 

'*''..// . • 

■ . ' ^1 

The next section examines the problem of developing Categories for 

• *• . . j!^ ' '" - - • ! 

placing project ^appl icants on a scale (STRONG, AVERAGE, and WEAK) for 

each of the impact statements used in th^ study. The LEA ratings of 

availability of data for each statement are also reported^. 
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•B. Predictive Impact Statements (Quest I6nna i re 2) 

' ' ' • ' • 1 ■ . \- 

Questionnaires 2 and 3 were designed tq^obtain information in 

two areas. The first area was thtf feasibility of collecting data on 
each^mpact statement, predictive or outcome. The second was the devel- 
opment of preliminary distribution data for each index to provide a 
first approximation to scale categories of STRONG, AVERAGE, and WEAK ; - 
that tan be used -in fundingrand evaluation decision making'- These two 
areas, provide the first step toward development"^ a rating scheme for 
projects that will have objectively assigned rati ng categories and can 
server to provi.de a total score for each project application or eval- 
uation .that Indicates the "^degree of impact (high or average or low) bf 
the" project. The data on feasibil ity an4 distributions for categories 
can be used by DOES in two-ways. The- first is to provide the basis' for 
further discussion of the implementation of " data collection by LEAs on 
each Impact statement. The second is to provide an initial set of cate- 
gories for scales that can be adjusted on the basis of data collected 
"on actual project applications and evaluations over the next few years. , 
Both Judgment and distribution data should,, be used in developing scale 
categories that are accepted and realistic to DOES and LEAs. . 

The data on feasibility and indices for each predictive impact state 

.. , . ^ • . . . ^ , . 

men t are presented below. The. indices have been grouped' i nto three cate- 
gories for the impact statement where possible,, given the distribution 
of the. data on the respective indices. There are moderate to severe 
limitations to the data reported below. The use of questionnai res to 
collect the distribution data has an inherent limitation in terms of 
communication with the respondent arid interpretation of the request for 
; foc/ciata. ' Some respondents clearly provided inappropriate data, and 



■. ■ : ' . : . 

^" ■• ■ •: ■*■ ■ .30 

data which did not meet a test of *Veason.ableness** were riot used for . 
the Fndex d 1 str ib^ution Involved. Also, the amount of data avaMable 
varied greatly from Impact statement to impact statement. Rather than 
impose an undue (and l ikely to be unmet) bur^Jen , respondents were in- 
structed to indicate the ava i 1 ab i 1 i ty of data where [t could not be 
quickly provided. A letter of instruction to LEAs requested that the 
fol lowing .process be used to review and provide data for the impact 
statements: 

a. filr out the questionnaire using the appropriate project 
as' 1 isted; 

b. for items requesting data that are not readily available' 
using the fiscal and program reports, do not attempt to * 

' complete the item. But, WHERE YOU OMIT DATA, PLEASE MARK 
EACH ITEM IN THIS MANNER: '\ ^ 

OK if the data could be supplied given advance notice or 
NA if the data are not accessibtfe without considerable effort or 
J_M if the data are impossible to collect,- ' ' ^ 

Respondents were requested to mirk an Q_K, N^ IM for each item left 

blank. Thus, in examining the data wh ich .fol 1 ow, coris^derable caution 
should be used . Al though the number of respondents for BOCES is high, 
the tables on data ava i 1 abi 1 i ty wil 1 usual ly indicate that considerably 
fewer BOCES (and large cities) supplied particular data. Also, the 
particular LEAs supplying data varies-H^m. question to question. With 
these caveats in mind, the fol low! n^data provide a basis for a, provi- 
sional . rating form, as summarized irj|, Part V. of this report. 

■ ■ ■ ■■ ■■ ■ . . ^ 
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ly Studi^nts. wi 1 1 bfi trained for occupations where jobs are available. 

Table 7 shows that a major! t^of BOCES and large cities could either 
supply the data on job openings. or indicated tijax the data could be 
supplied. Five.Boces and five large cities indicated' the data were not 
Table 7. Data . ava i 1 abl e on Job Openings 





Data 




Not. 


liiiposs- 


No. 






Supplied 


OK 


Access fble 


ible 


Response 


TOTAL 


BOCES 






•5 


. ^ 


5. 


39 


CITIES 


8 


f 


• 5 






20 : 



accessible without considerable effort, and one indicated the data ort 

job openings were impossible to supply. Since the predictive impact 

statements asked for data on a FY 1 979 grant appl icat ion , there is an 

apparent lack for some LEAs in the basic data that are needed to support 

a VEA application/ : 

The index developed for this impact statement consisted of the ratio 

Total number of local jobs available 
Tot^'al number of students expected to participate 

This index was computed for each of the 32 LEAs that suppl ied data and 

three categories were established ba§ed on the ratio above. The numbers 

of BOCES and Large Cities falling In each category are shown below: 



Scale 
Category 


No. of Jobs 
No. of Students 


BOCES , 


CITIES 


Weak 


•<i 


9 


1 


Average 


^ 1 < 2 


7 


2 


Strong 


> 2 


8 


5 



Intuitively, the funding applicant who shows that less than, one job per 
expected student is available Is a weaker applicant or Is less likely 
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.to .have an impact than one for whom at least one job and- less than 2 



IS. aval lable. Stronger applicants provide data that more than two jobs 

^ \ • ■ ■ ' 

are. avai lable in the local area for each trainee in the occupational 
program.- These .three categories can provide, the basis fo^"WE"AK, AVERAGE, 
and STRONG, program ratings. on this impact statement. The acceptability - 



of these categories can be checked with LEAs and the feasibil itsy of 



supplying data is very high. Data from succeedir^g years can be 



examined 



to. see whether changes in the labor market wa^rrant adjusting th^ category 
definition's. 

2. Project objectives are stated in measurable terms- 
Table 8 shows that a majority of BOCES and large cities canAprovide. 



a count of the numb'er of objectives .they have and the number that are 
stated in ''measu^bie'^. tern^i-^;^^^^^^^^^^^^^^ ! 

Table 8. Data available on project objectives- ! 





Data 




Not 


Imposs- 


No « 






Supplied 


■ OK 


Accessible 


ible 


Response 


TOTAL 


B,OCES 




1 


2 


1 


11 


39 




CITIES 


18 








2 


20 





The index computed was the ratio: 

' Number of measurable objectives 
Total' number of objectives 

The d /stri but ion of ,the index is shown below: 

• Scale No. Meas 



Cateqorv 


■ Total No. 


BOCES 


CITIES 


Weak 


0 


2 


2 


Ave ra ge 


>0<1 


8 


k 


St rtjng • 


, ■ r 1 


I'* . 


12' 



This objective' relies upon the judgment of the DOES staff to define 
measufabi lity . The defi nit ion of meas urab i 1 ity used in the FY 1979' 



funding, process can be used here: "two people can agree on what 
constitutes achievement of the objective." Projects which do not 
provide measurable objectives would be considered WEAK on this impact 
statement, those where the ratio was between zero and one (i.e., some, 
but not aM,, objectives are considered measurable) wculd be categor- 
ized as an AVERAGE project, and those where all objectives were meas- 
urable. would, be rated STRONG on this particular impact statement. 
3. No se.x discrimination wi 1 1 be made in recruiting and placing 
students in vocational programs. 

The data requested for this statement are checks of the activities 
in vocational education programsT Fhat will assist in sex equity. These 
activities are listed be.^ow, along with the percent of BOCES (N = 39) 
and large cities (N = 20). checking each activity. 

Check each activity below that will be carried out prior to or during 
this project . . ^ 

Percent checking: 
BOCES CITIES , 

85 100 Vocational courses are equally available 
^^^^^^^^^to^jfeiDaJ-e---an^ male students upon request. 

^5 Female and male students will be recruited 
in approximately equal numbers. 

' 7^ 100 All course instructional material will be 
free of sex bias and sex role stereotyping. 

28 25 Program teachers wi 1 1 equally represent 

females and males. 

51 65 *'Role models" of the "nontradi tiohal" sex 

for the occupation will visit the program. 

79 90 All career materials will be free of sex 

bias and sex role stereotyping. 

79 95 Men and women students are provided in- 

formation about their rights to equal 
educational and employment opportunities 
under the law. 



Percent checking: 
BOCES CITIES 
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69 



79 



69 



85 



85 
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A program will be conducted for parents 
which will assist them to work with 
their daughters and sons to consider all 
educational and employment opportunities.: 

Special support services or counseling 
will be provided to females and males 
who select a nontrad i t idnal occupational 
program 

Equal emphasis will be placed on financial 
support and cooperative educational placer 
ments for ferales and males. 

Program Instructional and related guidance 
personnel will be provided the inservice 
training necessary for the delivery of sex- 
fair instruction and counseling for . students , 



Table 9 shows that these" are very feasible data to collect, and the 
Table 9. Data available on sex discrimination checklist. 





Data 




Nof 


Imposs- 


No 






Suppi led 


OK 


Accessible 


ible 


Response 


• TOTAL 


BOCES 


33 


1 


1 




k 


39 


CITIES 


20 










20' 



Index that was used for this statement was: 

Number of sex d i scr imi nation i tems checked 
Total number of items (e leven) 



The distribution of the index is shown bel 



ow 



Scale 


No. items checked 






Category 


- 11 


BOCES 


CITIES 


Weak 


<.50 


k 


1 


Average 


J^'.so^.ys 


\ 


\k 


Strong 


> .75 


15 


5 



A first approximation of three categories was devised based on the ratio 
above, with a weak program (as defined here) checking less than half of 
the. possible activities to assure sex equity in recruiting and placing 



students In vocational programs. An -average program checked between 

i ■ 

half and three-fourths of thie activities, and a strong program 75 
percent or more of the activities. As indicated in the percents ' 
checking each activity, the most difficult ireas of activities are: 
recruiting approximately equal numbers of female and male students; 
having 'Vole models*' of the npntrad.it ional sex (for the occupation) 
visit the program; representing females and males equally as program 
teachers, conducting programs with parents to assist them to work with 
their daughters and sons to "consider all educational and employment 
opportunities (BOCES) , and providing inservice training necessary for 
the delivery of sex-fair instruction and counsel ing for students (large ' 
cities). 

4. Training objectives will be^met in. the most cost effective manner. 
Table 10 summarizes the data availability on program costs.* As 
Table 10. Data available on program cost. . 





Data 




Not 


Imposs- 


No 






Suppi ied 


OK 


Access ible 


ible 


Response 


TOTAL 


BOCES 


27 


1 


1 




10 


39 


CITIES 


' 18 








2 


20 



indicated, these data are typically available. The index for this state- 
ment was computed in two ways. The first was a direct ratio of cost to 
number of students, the second was a ratio of the cost per student to the 
percent of .program cost (estimated by LEA) to be covered by the VEA grant. 
The scale categories developed for each of these ratios are shown below. 

Scale 
Category 



WEAK 

AVERAGE 

STRONG 



Cost per student 


BOCES 


CITIES 


> $1000 


8 


9 


> $200 <$ 1000 


1 1 


8 


< $200 


8 


1 



4: 



Scale 
Category 



cost/studen 


\l% covered 


BOCES* V 


CITIES* 


> 

> $750 


$2500 
$2500 
$750 


7. / 


■ k 
2 



WEAK. 
AVERAGE 
STRONG 

• : *Some programs reporting costs did not; .est imate % covered 
The cost est imates used /in the ratios he^re incl ude .capi tal costs. Unless 
capital costs are incj^ijded, a majorlty'^f BOCES grants cannot have cist 
per student computed'. There were 29 BOCES grants reporting data on. 
cost and number of students in the program. Fifteen of these grants,' were 
for equipment only. The propbrtion of large city grants f<3r equipment 

only was lower:"^ three of .18 grants for which data are reported were 

• J 

equipment only grants. j " 

' ^ I- 

Adjusting the cosb per student ratio by the "percent of program! 

covered by the grant" provides a rough estimate of the total per pupil 
program costs, and the categories are for larger dollar amounts, as shown 
above. The dollar amount increases *by a factor of about 2 1/2 to 3. 
The change in ratip resulting from the adjustment by percent of program 
costs covered does not affect the BOCES distributions • into scale cate- 
gories. There are more large city grants where a smaller proportion 
of the program costs are covered by'VEA funds, resulting in a Change 
toward higher per student costs, and fewer "strong" projects on the 
criterion of adjusted cost per student. 
5. Large numbers of students will be trained. 

Table 11. summarizes the data availability on numbers of students > 
to be trained. As indicated, these data are readily available. The 
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Table 11. Data available on number of students 





- Data 




Not 


Imposs- 


No" 




• 


Suppi led 


GK 


Access 1 ble 


ible 


Response 


TOTAL 


BOCES 


27 


1 


1 


i; 


9 


39 


t 

CITIES 


17 








3 


20 



index for this statement was the frequency distribution of numbers of 
Students. Given the emphasis of the statement j the categories developed 



Scale 
Category 


Number of 


Students 


BdCES 


CITIES 


WEAK 




100 


17 - 


7 


AVERAGE 


> 1 00 


500 


5 


k 


STRONG 


:> 


500 


5 ■.. 


6 



here. show a skewed distribution for the BOCES. As the BOCES are gener- 

i ' . 

ally conducting sma 1 ler proigrams the scale categories are hot as approp- 
riate for them. However, the stale categories are shown here for Illus- 



trative purposes only and no at^tdcnpt was made to. change the category de- 
finitions for the BOCES. This impact statement received a low priority 
rating and would not be included in a rating scheme. 

6. Training will be provided to increase students' employment options.. 

Tahle 12 summarizes the data availability on employment options pro- 

vided by the vocational education program. The variable was defined by 

thi s instruction: 

Use the OE code to list the number and titles of occupational 
areas for which, graduates of this project are prepared (e.g., 
07.090^* Medical Asisistant, O7..O906 Health Aide.) 

.■.4 

As shown in Table 12, over half of the BOCES were able to provide this 
Information, as well as over three-quarters of 'the large cities. . 



Table' 12.. Data available on employment options. 



BOCES 

/ 

'^CITIES 



. - 'Data 




, Not 


imposs- 


No 




Suppl led 


OK 


Accessible' 


ib]e 


• Response 


. TOTAL 


21 


6 


3 




■ 9 


39 


17 


1 


1 




1 


20 



The distribution of number of employment options is shown below,^ and 



Scale 
Category 



Number of options 



1 



BOCES 



8 
8 
5 



CITIES 



9 



WEAK 

AVERAGE > 1 5 

STRONG ' . > 5 

indicates for this criterion of impact that there were a number of BOCES 
and cities (8 and k, respectively) who provide only 1 occupational title 
for which graduates of their program are prepared.. As is true of all 
the scale categories reported here, the range for .the average and strong 
categories may be too wide or too narrow. The categories should be re- 
exanriiled in 1 ight of the general expectations of the number of employment op- 
tions that it is r^sonable for a program' to attain. And, it is well to con- 
- sider that this impact statement i^s only one of s^ix or seven that might 
appear on a final version of a rating form;* Programs that appeared in 
a weak category here might well appear in strong categories on other 
statements. 

7. Students will be prepared to meet entry level. sR'i 11 requirements as 
specified by prospective employers. (e.g. employer * s rat ings of 
performance objectives In terms of job requirements). 
• Table 13 presents the summary of data 'that are available for this - 
impact statement. Slightly less than half -pf the BOCES could either ^ 
supply these data or could supply them with advance notice. Half of the 



large city respondents either had or could supply the data. 



Table 13, Data aval l.^ble on employers' skill requirements. 





tOata 




Not 


- Imposs- 


No 






Suppi !ed 


OK . 


Accessible 


ible 


Response 


TOTAL 


BOCES 


■ 12 




7 


3 


. 13 


39 


CITIES 


7 


3 ■ ' 


8 




2 


20 



The jtndex used to establish scale categories was,: 

Number of entry level skills checked by employers 

Number of entry level ski 1 1 s listed by school . . 

The scale categories could range from zero, none of the entry level skilj-s 

checked by employers, to one (1), the same number of entry level ski 1 1 s 

checked by employers as provided by the school. 



Scale 
Category 


No. 
No. 


checked bv emp lovers 
listed by schools 


BOCES 


CITIES 


WEAK 




< .70 




0 


0 


AVERAGE 




.70 < 1 




2 


0 


STRONG 




1 




10 


7 , 



These data must be considered very tentative, since so few respondents 
could supply data. Also, there was a wide range in the number of skills 
that BOCES particularly said had been listed for- the program and/or 
employers. BOCES listed from ]k to well over 500 skills as entry level 
skills. The range for the cities appears more realist^.c,, from 3 to 30^ 
entry level skills* The implementation of this Imp^t^^i 'sft^atement will 
apparently require more effort . in defln^mg ^nhtt^i^ le.^c^ skills and 
contacting employers to determine the overlap between BOCES skill re- 
quirements ajnd employer skill requirements. 
8. Program will serve- students * i nterests . 

• The data for this .statement were to be based on a source such as 



a career 1 interest, survey.. Respondents were asked to Indicate: 1. . th^ ' 
total number of students surveyed ; and 2. the -.number of students 'select- 
Ing this (th6 program) occupational area* as first or second choice (of 
Interest to them) . Table 1 if presents the availability of data on. student 
Interest in the program, occupation* . 

. Table IV. Data- avai lable on students' interests 





. Data 
Suppl led 


OK 


Not 
Accessible 


Imposs- 
. Ible 


.No... 
Response 


TOTAL 


BOCES 


9 


9 


7 


2 


12 


: 39 


CITIES 


7 


9 


4^ 






'20 



As the table Indicates, few BOCES or large cities were able to supply 
these data, so no, Index'is presented here. . However, an index which can 
be used is": . , 

^ Number of students to be served by program 

' Number of students selecting area first, or second ■ 

An examination of the fevy grant' appl icants supplying these data showed 

that ratios obtained for BOCES programs were: 1.5, 2.6, 1, and' 1.7v That 

is, there were at least' as many interested studecjts for enrollment as 

there were program places. For the large cities the following data were 

available: 2.k, 1,1, 2.1, and .78.' Only in the last instance, apparently, 

were there fewer Interested students identified than program openings. . 

Possible scale categories are: ' 

Scale Category No. In'proqram- 
; " No. Interested 



WEAK " 

AVERAGE 

STRONG 



•<-l 
>1 4^2 

>2 



, , Stnce/-f^\ LEAS were able to" supply data, but at least as" many: said , 
• / data could-^beu^^pl led wi th advance notice, i t would. be; important to , 
.carefully define the procedures for a survey and' the data being requested. 

While some LEAs Indicated that these, data were most appropriate for new 
, progranis. they may also be importantfto justify request^for program 
Improvement and especial !y equipment grants. This appeacs.to be im- 
portant in l ight of the fact that a minor i ty. of .grants are given for. new • 
_ programs, for -the. LEA(ll ;of the 39 BOCES grants. 30%; and 5 of 20 large- 
cities grants. 25%). For the program Improvement grants. 23 of 25 to 
BOCES were grants with an equ i.pment. emphasi s— equipment costs 
were 50% or more of the total grant cost. . For the "large cities, 6 of 11 
program -.improvement grants had a similar equipmentUmphasis. . 
9. Program is articulated with local post secondary institutions: . 

The data available on program artrculation requested were the numbers 
^bf.local post secondary, institutions where students could continue in the 
san^ Occupational field. The data ava f la'bi 1 i ty a. /presented inTable'l.5. 
Table 15. Data ava i lable on articulation . 

•No 



\''f^^ Not ..Imposs- 



BOCES 
CITIES 



•21\ 


3 


' 2 


1 U 1 c 

' 3 . 


Response 
10 


TOTAL 
. 39 


18 - 


\, 1 




1 




20. 



The index that was examined for scale categories ^was the number of local 
post^secondatry institutions. Tentative scalecategories are given below. 



WEAK 
[ AVERAGE 
STRONG 



No. of institutions ' 


' BOCES 


' CITIES 


0 . 
1 - 2- 
3 or more .- 


■ 5 
9 ' 


1 

if 
13 



The BOCES have nx)re dlf f Icul ty wl th tPKJs Impact "statement than the large, 
crtles. Seven BOCES rfepprte.d no articulation with a post secondary 
institution, while onJyl large city report^!- no articulation. However, 
this is one of the impact statements that received a low priority, so • 
the disparity and possible Inequity because of \he location of particular; 
LEAs will not be of- concern If this statement is \ot used as part of a 
funding rating s'cheme. ■ " 

10. Program will be repllcable In other LEAs. 

The Information requested, in this statement was a check of the 'Mtems 
in a .program that could be reproduced and sent to another school. The Items 
checked and the percent checking each ivtem were: \ 



Percent checking: 
BOCES CITIES Item 

77 95 course outline 

. 50 \ lesson^^plahs * 

;i.3 55 ^ \project-<l^le 1 oped student' workbook/ 
' \ student I nstructlohal materials 

62. 90 Istudent assessment forms (tests or 

y rating forms) 

21 25 project developed si Ides/AV materials/ 

f 1 1ms ' , 

72 100 list of recommended equipment/textbooks/ 

AV "material s • 

67 9.0 -list of suppliers and vendors 

33 . 55 . teacher program guide 

77 ■ 100 project proposal 

62 [90. V program evaKiation report—data and 
s'ample forms- " . 
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. The- f tennis for rep 1 i cab! 1 I ty,:that^^^w^ least frequently avai lable we're: 
lesson plans / pro|ect-deve loped student workbook/studen^^ * nstructional 
materials; siides/AV/films, and teacher-^ guides. The numbers of 

programs reporting this information is gi,ven in Table^ f6, ' . 

Table 16. Data available on repl I c^bi 1 i ty, • , 





Data 
Suppi led 


OK 


Not 
. Accessible 


Imposs- 
ible 


■ No~ 
Response 


TOTAL 


BOCES 


31 


' 1- 






■>f - ..^ 
. 6 


39 


CITIES 


20 










~^ 

20 



All of the Jarge cities and a majority of BOCES coUld readily supply 
the information. . 

The. index for which scal'e categories were developed is: 

. Number of 1 tems- checked as being replicable 
Total number of iteims 

The scale categories for^the index are shown below: 

o ■ ■ 

Scale- 
Category 

WEAK 

AVERAGE - 
STRONG 

The index has a'-range from 0 to 1.0, and a majority of the programs re- 
porting data would be classl.^iied as average or strong using this type of 



-No. of 
Total 


items checked 
no. of i tems . 


BOCES 


CITIES 








.8 


4 


> 


.5 <.8 




o 1 3 . 


7- 




>.8 




10 


9 



index. 

In addition to developing indice.s for the predictive impact statements, 
the survey included another questionnaire. Questionnaire 3. which was in- 
tended to provide the. basis for developing similar indices for the outcome 
Impact statements. The results of this analysis are presented in the 
next section. . • ' ■ 



,C. Outcome Fmpact Statements. (Questionnaire 3) I - 

Xhe outcome statements- are evaluation data thar'would .b2 sub- ' <: 
. mi^ed as. part of the flnaT project report and evaluation, "he. outcome- 
and predictive impact .statements are parallel in content^ 1 though . 
differing sometimes in the exact type of information to be presented. 
Thus there may be the sSme index for the predictive and outcbme state- 
ments (as in the replicabiHty statement) or the index may- be different," 
as in the dase of the sex discrimir%ation statement. The indexes for the 
outcome statements are presented below, along with the information on 
data availability. ' . ' 

Outcome Impact Statements - 

- " Program graduates are working, in occupations' for which they were trained, 
Several of the outcome statements are based on a program ■ , 

graduate survey (see Appendix D, ■ Questionnai re 3, for the form) Overall, 
28 BOCES (of 3?) and ]k cities" (of 22) suppl led^some data on the form. 
However, the numbers showing "data supplied" will vary between impact 
statements since n9t al l LEAs responding gave the same information. 

Table 17 summarizes the ^data availability for outcome- impact state- 
ment 1. A majority of bo.th the BOCES and large' cities indicated that 
Table. 17.. -Data available on^graduate employment 





Data 
Supplied 


OK . 


Not 
Accessible 


Imposs- 
. ible 


No 

Response 


TOTAL 


BOCES 




3 






12 


39 


CITIES 


10 i 


2 


2 


2 


.6 


22 



the data were supplied or could be suppl led with advance notice. The 
large number of "no responses" probably'indicatfes that these LEAs do not 
routinely follow-up program graduates. 



The index for this statement is : ' ' , 

Number empl oyed in field for which^trained 
. Total available for full time employment " 

The base for the index is the total number cf prograV gra. uates avail- 
able; for full time. employment. Excluded are those lookfnc for part time 
employment and those not available for full time employme • because 
they are attending school are in .mi. lltary.servi.ee, or^^ homemakers. 
The index ranges from 0 to 1, and the scale categories ar^ shown below. 



Scale 
Category 


No. Employed 
No. Available 


BOCES 


CITIES* 


WEAK 




9 


k 


AVERAGE 


>.5,^ .75 




- k 


STRONG 


>.75 




2 



The scale categories shown are arbitrary, and should be considered hn light 
of what are considered satisfactory pi acement and employment rates for 
vocational educat iVn programs. Further col lection of data on placement ' 
rate experience for\,BOCES and large cities would be' useful in establ ishing 
scale categories for this index. If the data are col 1 ected rout inel y, 
and scale categories established on the basis of experience, weaker programs 
can be- identified and assisted to improve their prog ramor placement efforts 
2. Project objectives are fulfilled 

Table 18 summarizes the aval labf U ty of data. for th is .objective. 
The questionnaire asked for the number of measurable project objectives 

listed In the proposal and the number of measurable project objectives 

I ... . ^ 

met. . A "measurable objective'! was defined as. a project outcome that can be" 
Stated as a count, proportion, percentage, or another quantity. A 
large maj.orlty of LEAs can supply the* information. 



Table 18. Data available on project objectives 



s 


Data 

Suppi led 


OK'. 


• Not - 
Accessible • 


Imposs- 
ible 


.No • 
Response' 


TOTAL 


BOCES 


26 ' 


2 


k 


1 


6 


39 


CITIES 


21 


1 








.22 - 



The iindex for this statement is: . 

' . Number of measurable objectives met 
Total number of measurable objectives 

The scale categories are shown below, and indicate that a majority 



Scale 


. No. Meas\ Obj. 


Met 






Category 


Total No. meas. 


obj . 


BOCES 


CITIES 


WEAK 


0 . 




' 3 


0 


AVERAGE 


> 0 1 




3 


.3 


STRONG 


1 




'20 


18 



of LEAs can meet this outcome impact statement. One difficulty with this 
Index Is that there Is no Ind^'icatlon of the type of objective that is 
met or not met. However, the use of the Index may help to strengthen 
the listing of objectives and the specificity of project planning. . 
3. No sex dUcrimlnatlon occurred In student selection, training and 
job placement. ■ " 
The data for this objective appear, on the program graduate survey 
form In Questionnaire 3. The data on /initial enrollment, graduates, 
employment, and so on, are collected for females, males and total group. 
Somewhat fewer programs were able to supply these data.jthan the data for 
statement 1 . The information on data ava I lab 1 1 I ty are in Table 19. 





Table 19 


. Data 


aval lable on 


sex discrimination 




1 


Data 






' Not 


Imposs- 


No 




if 1 


Suppl ied 


OK 




Accessible 


ible . 


Response 


'TOTAL 


BOCES 
CITIES 


22 


5 


1 


0 


11 


39 


9 


2 . 


k - \ 


1 


6 


22 



j Program Total Program Total [ 

IF there are both females and males in the program it is possible to 
compute what are labelled below as index B and Index C. If there only 
males or only females in the program, indexes B and C are not meaning- 
ful . The scale categories for Index A are given below: 





Scale 
Category . 


c 

Index A 


BOCES 


C ITIES 




WEAK 


1 


. 13 


0 . 




AVERAGE 


< 1^.25 


7 


k 




STRONG 


<.2S 


2 


0 



As indicated by the values for the scale categories, an index of VM" 

means that the program enrolled either all males or all females. The 

further the index is from 1, the more equal is the numbers of females 

and males enrolled. 

Index B consists of the following ratios: 

INDEX B.: [ Number of female graduates - minus Number of male gradi 
I Number of females enrolled Number af males enn 

The scale categories for Index B are shown below and, as indicated, the 

smaller the difference between the two ratios, the less discrepancy betwe< 

the numbers of females and males enrolled and graduated. 



Scale 
Category 


Index B 


BOCES 


CITIES 


WEAK 


^35 


k 


-1 


AVERAGE 


<.35^.10 


2 


0 


STRONG 


<.io 


■ 3 


3 



^ ..-^jOffe. — - 
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Index C Is based on the ratios of females and males employed t(|) those 

available for full time employment. Index C Is computed as follows: 

INDEX C. Number of females employed minus Number of males employed 
Number of females available Number of males aval lable 

V 

The scale categories for Index C are shown below and the Interpretation 
is similar to that for . Index B: The smaller the difference between the ' 
two ratios, the greater the agreement on the proportions of females and 
males finding employment- 



Scale 
Category ^ 


Index C 


BOCES 


CITIES 


WEAK 


' " '^ .35 


1 


1 


AVERAGE 


< .3S?.10 


h 


0 


STRONG 


< .10 


^ 3 


2 



The data shown In the three indexes Indicates that there are successively 
fewer programs that can supply the data, that oer half of the programs 
reporting data enrolled only females or males, and that there appear to 
be program differences In placing males and females using the scale 
categories devised for the indexes B and C. 

k. Training objectives are met in the most cost effective manner. . 
This outcome statement is the same as the predictive statement, 
, except that the outcome statement is based on the actual VEA monies 
expended on the grant. Table 20 summarizes the numbers of LEAs providing 



data . 



Table 20. Data ava 1 lable on cost per student 





Data 




Not 


Imposs- 


No 






Suppl ied 


OK 


Accessible 


ible 


Response 


TOTAL 


BOCES 


30 s 


2 


2 




5 . 


39 


CITIES 


22 










22 



As Indicated, these data are readily available. The Index here Is the 



cost per student, obtained by dividing the total grant monies expended ■ 
by the number of studei^^i^^ved by the grant. The le-^s the amount of 
money spent per student, the "stronger" the program is on this criterion 
The scale categories are shown below. , ^ 



Seal e 
Category 



Cost per student 



WEAK 

AVERAGE 

STRONG 



BOCES 



9 

12 
9 



CITIES 



2 
1 1 
9 



>'$1000 
P$200 4$1000 
<. $200 

As with the predictive impact statement on cost per student, a second 
index has been computed. The data available for this index are shown 
in Table 21. The adjusted index taKfS into consideration the percent 

Table 21: Data available on cost and percent covered. 

Data Not Imposs- 



No 



TOTAL 



BOCES 
CITIES 



;)upp 1 1 ea 
2k 


3 


3 




• 9 


39 


20 








2 


22 



of the project /prog ram costs xoveredb'y the grant and fs computed by 
using: 

cost per student 
percent program covered by grant 

Since programs differ in the amount or percent of the costs covered by 



grant, the 


category scales are d" 


f ferent 


and the project 


shift . 






\ 


Scale 
Category 


Cost per student/ 
% covered by grant 


BOCES 


CITIEs\ 


WEAK 


>$2i)00 


12 


■ 1 \^ 


AVERAGE 


>$500 <<1$2000 


9 


li» ■ 


STRONG 


<$500 


3 


5 



Different programs would be identified as "average*' or "weak" and as 
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having higher program costs per student,. as opposed to higher per student 
costs in grant monies only. It is arguable which index is preferable* 
for evaluation data, and perhaps both should be examined. 
5. Large numbers of students are trained. 

The data available for this outcome statement are presisnted in 
Table v22. 

Table 22. Data available on numbers of Students., 





Data 




Not 


1 mposs- 


No 




Suppl ied 


OK 


Acces's ibl e 




Response 


TOTAL 


BOCES: 


31 


2 


1 


5 




39 -. 


CITIES ^ 


22 










22 



The index is the number of students and the scale categories are pres- 

. I ■ 

en ted below:/ 



Scale 



Categories 


No. of Students 


BOCES 


CITIES 


WEAK 


^ 100. 


23 


8 


AVERAGE 


> 100 ^300 


3 


5 


STRONG 


> 300 


5 


■ 9 



As shown, the scale category distributions differ for the BOCES and 
large cltie.s. There is no satisfactory way to set the scale categories 
in isv^lation from other variable.s, and since this statement received a 
low priority ranking, i t shoul d probabl y not be used in a rating, scheme. 
6. Training increases student employment options. 

This statement requested the OE code numbers and titles of occ'upa- 
tional areas for which graduates of the particular program were prepared 
and the statement is comparable to the predictive statement for this 
variable. The numbers of programs supplying data are shown in Table 23. 
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Table 23, ij)ata available on OE employment codes. 





Data 




Not 


Imposs- 


No 






Suppl ied 


OK 


Accessible 


ible 


Response 


TOTAL 


B.OCES 


. 23 


k 


... :.3.. 




9 


39 


CITIES 


'1-6 


1 


i 1. 






22 



As shown, a majority of LEAs indicate that data are available or could be 
supplied. LEAs should probably be supplied with a list of the OE codes, 
if they do not have them,^ The index for the statement is the number of 
employment options (codes) listed. The scale categories are shown below. 



Seal e 
Catfegory 


No. of Options 


BOCES - 


CITIES 


WEAK 


.1 


k 


6 


AVERAGE 


"> 1 3 


. 5 


3 . 


STRONG 


>3 


li» 


7 



The scale categor ies def ined a "weak" program on thi s statement as a program 
for which students were qualified for only. one OE employment code. "Strong" 
programs were identified as those providing students with. the option of 
seeking* employment in three or more OE occupational codes. 
7. Employers are satisfieid with graduates of program. 

There were few programs who supplied data on this statement and about ^ 
half of the LEAs rated this request as data not easily accessible or as 
impossible to collect. However 13i 33% of the : BOCES ei ther supplied data 
or indicated it would be OK to supply the data given advance notice. The 
data are summarized in Table 2^. 

Table 2^. Data avai fable 6n employer survey. 





Data 




Not . 


Imposs- 


No 






Supp 1 ied 


OK 


accessible 


ible 


Response 


TOTAL 


BOCES 


9 


k 


9 


3 


\k 


39 . 


CITIES 


1 ■ ; 


1 


^ 1 


3 


9 


22 
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The Index, for! this statement was computed on the smaM amounts of 
data that were submitted. The index was: 

Number of students rated by .employers as Excellent or Good 
Total number of students rated by employers 

The distribution and the scale categories are shown below: 



Scale 
Categories 


Exc . or "Good 
Ratings/No. Students 


BOCES 


CITIES 


WEAK 


<".5 


0 


0 


AVERAGE 


> -5 < .9 


3 


0 


STRONG . 


Z -9 


6 


1 



These scale categories should be discussed by DOES supervisors, since so 
few BOCES and only 1 -large city reported data for the statement- 
8. Students trjained have positive attitudes toward work. 

The question that suppl ied data for this statement appeared in Quest- 
ionnaire 3, the Program Graduate Survey page. .The data were based on students 
giving rati(ngs of job satisfaction, one indicator of positive attitudes toward 
work. Three categories of ratings were assumen: Excellent/Very Good; Good/ 
Average; and Fair/Poor. The data available to look at the index categories 
is presented in Table 25. ■ / 

. Table 25. Data on student work satisfaction 





Data ' 
Suppl ied 


OK 


Not ■ 
Accessible 


Imposs- 
ible 


No"^ 
Response 


TOTAL 


BOCES 


12 




5 




16 


39 


cities' 




2 


5 


3 


8 


22 



The index calculated' for this statement was: 

Number of students rating job satisfaction as 
Excellent/Very Good and Good/Average 
Total number-of students rating their job satisfaction 



/^galn, a small: number of BOCES arid large- cities were able to supply data 

/.■•.•■ 

on this statement. The scale categories based on th6se small numbers of 



' LEAs 



are below. 


! 






Sea 1e 
Categories 


Job Sati-sfaction 
Index 


BOCES 


CITIES 


WEAK ' ' 


.90 


k 


0 ' 


AVERAGE 


> .90 * ; 1 .00 . 


3 


. 1 


STRONG' 


1 .00 


■ ; 5 . . . . 


3 , 



As Indicated by'the scale categories, there were few proportions below .SO. 

Over half the proportioxis were either 1.0 or between ,90 and 1,0, For the 

sample available here, most students report themselves as average or above 
' in satisfaction wTth their jobs. If this information is desired as part 

of the final evaluation report for LEAs, effort should be devoted to devel- 
. oping a common fol low-up form for students (and for employers, for state- 
. ment 7.) * 

9. Students tra-ined continue their education. 

The data for this statement again appeared in the Program Graduate 

Survey page in Questionnaire 3. Four categories were listed for students 

continuing thei r education: Technical School-continuing in same field; 

and Coliege-ln different field. The data available for this statemenV'a^^^ 

. presented- in Table 26. 

^ Table 2$. Data available on continuing education 





Data 




Not 


Imposs- 


No 






Suppi led 


OK 


Accessible 


ible 


Response 


TOTAL 


BOCES 


15 


•5 


2 




17 


39 


CITIES 


8 


2 


3 


2 


7 


22 



The index for this stateiDent used two of the four categories for con- 
tinuing education — technical school and college In the same field. 



Number of students continuing In same field (technfical school and College) 

Total number pf' program graduates 

The scale categories for this index are narrow fn ranges since not all 

program graduates continue education. The scale categories are shown below: 

\ Scale' 
Categories 



No. Students/No. Grads'. 


BOCES 


CITIES 


■ ' -V 


7 


k 


>.i .k 


5 


3 




' ^ 3 ■ 


1 . 



WEAK 
AVERAGE 
STROf^G 

Although this index can be used, it did not receive a high prior i ty rating 
or ranking^ so it is unlikely to appear on any final rating of Project 
impact . However, continuing education is a legitimate VEA outcome and acknow- 
ledged In the 197.6 leglslatron, so LEAs and states record keeping need to ' 
make provision for record keeping in this or a" similar set of categories, 
to. Program can be replicated in other LEAs. 

The .data for this' statement were the number of items that the LEA. 
checked thatcould be reproduced and sent toanother school. -The items 
are listed b^low, along with the percent of BOCES and large cities responding 
for each item (that it could be sent to. another school). 



Percent checki ngs 
BOCES \CITIES 



82 
kk 
26 

77 

23 

7k 



V 
6i» 
36 

64 

' 9 
82 



I tern ;■ ■ . 

course outl i ne 

lesson plans - 

project-developed student workbook/ 
student 'instructional materials 

student assessment forms (tests or 
rating forms) 

project-developed slides/AV materials/ 
f i 1ms 

list .of recommended equipment/textbooks/ 
AV/materlals 
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Percent checking: 
BOCES CITIES 



77 
33 
82 
72 



82 

95 
77 



I tern \^ 

list of -suppliers and vendors' 

teacher program gui.de 

\^ 

project proposal \ 

\^ ' . . 

program. eval uat Ion report-data 
and sample forms \ 



The Items that were least frequently available for reproduction for 
another school were: lesson, p.l ans ; project-developed student work- 
book/student instructional materials; project-developed slIdes/AV 
materials/films; and teacher program guides. The numbers of LEAs 
reporting data aval labi 1 i ty are shown in Table 27. ; . 

Table 27. Data aVal lability on program replication Items. 



1 


Data - 




Not 


Inrtposs- 


No 






Suppi led 


OK 


accessible 


; ible. 


Response 


TOTAL 


BOCES 


•31 


1 


1 ■ ■ 




6 


39 


CITIES 


21 








. 1 


21 



These data^re read! ly aval lable from LEAs. . 

The Index for this statement Is the same as the predictive impact 
statement : ' 

Number of Items' checked 
Total number of items 

The scale categories are shown below ' 



\ Scale 
Categories 


No. checked 


BOCES 


CITIES 


WEAK , 




7 


6 


AVERAGE 


.5<..8 


]k 


S ■ ' 


STRONG 


. > .8 


10 


6 



As shown by the categories, well over half of the LEAs checked 5 or 
nx)re of the statements, resulting in proportions of ,5. or more. 

The next section of this report presents a sample rating scheme 
that incorporates the predict iye impact statements along with other 
- DOES categories used in FY 79/ Further comment on the outcome (eval 
uatlon) statements appears In the Summary , 



: V, Recommended Rating .Form: PredictiVe Impact Statements 



The predictive impact statements that Were given the highest 



.ranks and ratlings were seven in number, further check 



tification of the seven highest statements was made by deriving weights 



.of the i hden- 



data as^ 
(Guilford, 195^). 



for the seven variables using a nfiethod for treating , rank 
though a paired compari son seal ing had been carried out 
the weights were derived using the rank ordering given bj/ the DOES 
supervisors, (The weights could have been derived>from jthe total sample, 
but the computational time involved was not considered justified. The * 
set of five ranks provided by the DOES supervi sors was a pract~rcal, com- 
promise, since there was satisfactory agreement among all three sets 

of ranks 'provided by the BOCES, large cities, and DOES supet-vi sors) . 

• * ■ ■ 

^ actual procedures used for computing the weights for the seven 

predictive Impact statements is given in Appendix G, along with the 
description of the method and the weights for the outcome' impact statej;;' 
ments. The scaled weights were , approximated by using whole number 



weights of 3 f 2, and 1, FigQre 5 presents a recommended 



rat- 



ihg .-mrm for Project Qua 1 ity Critg.rja^' Pmpa.c.t" , . Jh Figure 5 are the 
seven predictive im^^t statements given "the highest prior i ties, the 
index developed for ea^^i statement, the scale category values for the 
three categories of Weak , ^verage , and Strong, the we:ight for each 
statement, and a column to record the scale category value of 1 (Weak)^, 
2(Ayerage), and 3 (Strong) multiplied by the statement weight. The 
total "impact score" for a project proposal can rangie from 11 (ratings 
of weak on all impact statements) to 33 (ratings of strong oh all state- 
V_ ments multiplied by statement .weights) . 

ERIC 



As part of the process. of developing the impact rating form, 
the FY 1979 DOES screening criteria and project quality rating y 
forms for management and instruction were also reviewed, (Appendix 
A). This revievy provided some further suggestions for considera- 
tion in any contemplated revision of the DOES rating procedures. 
A review of all three documents together led to these suggest ior^s : 

a. Some economic manpower information screening criterin should 
be placed on the checklist with the Initial Project Screen-. 
ing Criteria, rather than being rated: 

T, Information on labor market needs is directly related 
to the' specific program seeking funding under VEA. 
(Yes Q or No □ ). 
ii; In-fomation on labor market needs deals specifically 
with needs of* the region or area served by the educa- 
tional agency. (Yes Qor No Ql ). " 
J i i Information on labor market needs is based on objec- 
tive data and is corroborated by at least two data 
V. scjrces. . (Yes □ or Noj~],). ..- 
. b. The f i rst. econjomic/manpower needs screening criteria (#1) 
'*The project serves an area of high economic need as evi- 
denced by high rates of unemployment, concentration of 

» 

low income families,...'* can appear on the proposed (see 

below) quality ratiog form for Management/Planning. 

. -"^ ■ ■ . ■ ■ ■■ ' 

c. Some statements that appear for rating now in the manage- 

s • .. . . 

meht and • instructional quality forms should appear in the 



screening criteria or In second level screening on 
I manageifnent' and Instructlon/equlpment; 

- d. The remaining statemeats" in the management quality and 
Instructional quality forms should be divided "into two 
parts : 

Part 1: a checklist of Items that are criteria, the. 

project should meet or be revised/rewritten 
to meet (e.g. , instruct ional /cur ricul um, 
objectives are achievable within the dura- 
tion of the project) ; 
Part 2: a set of rating scales that call for . j udgments 
' of the quality of the item being rated (e.g., ^ 

How important are the project objectives? 
_ ' ^ ' Highly Important, average in importance, of 

low importance? AND How qualified are the 
project staff? highly qiial i f led ,. average in 
qualifications, below . average, in qualifications? 
, / AND What I s the* qual Fty of the past performance " 

of agency management? Excellent management, 
average in management skills, below average- in 

management skills? . . * 

If- the last suggestion above is examiijed, qual i ty rat i ng scales 
for Management/Planning and Instructional/Equipment might have the 
rating scales suggested in Figures 6 and 7- The division of the 
present forms into the "check! i st and ..quality rating sciales may assist 



* FIGURE 5 ' 
Project dualtty Criteria; Impact 



Impact Statement 



1. .Students will be 
trained for occupations 
where jobs are. available, 



2. Students will be pre- 
pared to meetentry level 
skill requirements of 
\ employers. 



3. Project objectives are 
stated in measurable terms. 



k. Training objectives 
^ will be met in the most 
cost effective manner. 



Scale Categories 
WEAK ' . kvERAGE 



STRONG 



WEIGHT 



SCALE 
X 

WEIGHT 



Total number of local Jobs 

Total number of program students 

1 . 2 . V 

No. entry level skills checked 
No. skills listed by school 

^ 1 

□ 

3 

No. measiirabl e objectives 
Total no. of objectives 




Employer 



<.7 IJ<\ 

□ 



a 

1 



0 ^Q^l 

□ □ 

1 2 

Total . grant $ 



1 

□ 
3 



X 2 



Total no. students served' 
> $1000 >$2004$ 1000 ^$200 

a a- ■ □ 

1 2 - 3: ; 



] 



.5. No sex discr iminat^ion 
will be made i rr rec'r u i t i ng" 
and placing students in 

• voca t i ona 1 p rog rams . r ' 



6. Trair)i.ng will be pro- 
vided tO; increase students' 
employment options. 



7.. Program will serve 
students ' interests. 



No. 'sex discrfmination items checked 
Totalxno. of items to check 

1 ,2; 3 X , 1 

Number of employment options (OE code) 
1 ^1 5' 

□ □ _ 
1 2 ^ XI 



3 



Number of students to be served 

No. students selecting area 1st or 2nd 

-dl >142 >2 

n □ - □ i 

1 2 3 - X 1 



Total Impact Score: (Maximum 33) 
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• FIGURE 6 

Project Quality Criteria:' Management/Planning 



Area 

1. The project serves 
an area of high economic 
need as evidenced by high 
unemployment and high con- 
centration of low income 
fami 1 ies. / 

2. The project serves a 
high proportion of handi- 
capped students among 
program students. 



3. The qual ity of project 
objectives in relation to 
similar projects is: 



k. The management and 
planning activities of the 
project, compared to 
similar projects is: 

5. The qual i ty of the 
management personnel for 
the griant is: 



Scale Categories 

Level of unemployment and low income 



Rat 



Average Above average 
fur region for region 



□ 

2 



Below average 
for region 

□ 
1 

Proportion of handicapped served in 
relation to similar agencies 

Below average Average Above average 
for region for region for region 



□ 
1 



□ 



□ 

3 



Below average Average Above average 



□ 
1 



□ 
2 



□ 
3 



Below average. Average. Above average 



□ 

1 . 

Below average 

□ 
1 



f). The quality of the eval -Below average 
uation plan, compared to j j 



similar programs, is: 



1 



7. The quality of the , Below average 
agency management and plan- 
ning in past performance I I 
has been: ' 



2 

Average 

□ 
2 

Ave rage 

CZ! ' ^ 

2 

Average 



9 



□ 
3 

Above average 

□ 
3 

Above average 
3 

Above average 

□ 
3 



Total possible (unweighted) points = 21 



6L} 
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♦ FIGURE 7 

Project Quality Criteria: instructional/Equipment 



Area 

1. The importance of the 
instructional objectives 
of the project is: 

2. The quality (qualifi- 
cations) of * the instruc- 
tional staff is: 

3. The qual i ty of the 
curriculum for this occu- 
pational area is: 

k. The life expectancy of 
the equipment/skills the 
project proposes to pur- 
chase/develop is: 

5. The facilities avail- 
able or proposed for. the 

i nstruct ional program/ 
activities and student 
populat ion are: 

6. The costs of the staff/ 
Instructional material s/ 
equipment. (in relation to 
costs for similar occupa- 
tional prograjp*') are 

7^^^^J>r€qual 1 1>^ of the 
Sgency^s past or present 
instructional programs is: 



Scale Categories . 



\ 



Rating 



Below average Average Above average 



□ 
1 

Below average 



1 \ 
Below average 

□ 

1 ' . . 

Below above 

□ 
Below average 

Above average 
.-costs 

□ 
1 




Lower than 
average costs 

□ 
3 



Below average Average Above average 



□ 
2 



□ 
3 



Total possible (unweighted) points = 21 
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in clarifying the nature of the rating task, as well as emphasizing 
to lea's that there are minimum standards suggested by the check 
list that must be iret. In addition, project appl icants wi 1 1 know 
that there are qualitative standards which will be applied to the 
proposal in the areas of management/planning, instructional/equip- 
ment,. and "impact," — the predicted effectiveness of the project on 
variables related to highly important outcome variables. If there 
is a- decision to use Quality rating scales in the two areas suggested, 
then it would be possible to establish weights for the statements 
within each area using a method similar to the one for the impact 
statements (perhaps using only the group of supervisors* rankings). 
It may also be desirable to weight the three areas' total scores « 
equally as can occur with the sample forms presented which contain 
the same number of statements for each of the three areas (7),* as 
well as the same number of score points alloted to each scale 
category — 1,2, or 3. There only need to be weights for the 
statements or weights for the total scores. The summary of the 
project and of the recommended next steps are given in the next 
section* 



Vl, Summary and Next Steps 



Summary' 





During 1976-77 the Department of Occupational Educat^ion*^ 
Supervision funded a study to develop definitions of p^ject 
"Impact" and to relate these definitions to funding decisions 
made by DOES. Preliminary statements of Impact were defined and 
ratings of priority assigned by DOES supervi sors- /^In 1977-78,. the 
second year c5f the project has concentrated of f u[tbep--ref'rhement- 
of the Impact statements, both predictive and outcome, In line with, 
the 1976 Amendments _(PL 9^-^82). In addition, It was 'desired to 
collect data related to both the "validity" of the Impact statements 
for funding decision making and evaluation of VEA funded projects, 
and the "feaslbl 1 ity" of collecttag data on the impact statements. 

Validity for the Impact statewents was defined in terms of the 
agreement among DOES supervl sors , BOCES d I rectors , and large cities 
project directors in their rankings and ratings of the Importance 
of the predictive and outcome impact statements- Feasibility was 
defined by asking- LEAs to complete questionnaires for two sample 
(but real) projects that would provide data on how readily available 
certain pieces of Information were or how difficult it might be to 
provide the information. One project that had been completed in 
1976 Was Identified for each BOCES and a minimum of five for each 
of the large cities (with 10 for New York City), and similarly 
project applications for FY 1979 funding were identified. The 
LEAs completed three questionnaires altogether, and the DOES 
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supervisors completed one--the ranking of importance of the predic- 
tive and outcome impact statements. 

The outcome impact statements that received the highest rank 
order and ratings were: 

Students trained have positive attitudes toward work 
Training increases student employment options 
Employers are sat i sf ied wi th graduates of program 

Pptigram graduates are working in occupations for 
which they were trained 

Project objectives are fulfilled 

Training objectives are met in the most cost effective 
manner 

The six predictive statements that received the highest rank 

order and ratings were: 

Students will be prepared to meet entry level skill 
requirements as specified by prospective employers 

Training will be provided to increase students^ employ^ 
ment options 



Students will be trained for occupations where Jobs are 
available 

Program will serve students* interests 

Project objectives are stated in measurable terms 

Training objectives will be met in the most cost effec- 
tive manner 

Questionnaires 2 and 3 provided the data on feasibility and 
also the sample distributions of data for the indexes developed for 

each of the predictive and outcome impact statements. For the 
outcome impact statements there were LEAs who had provided or would 



be able to provide the data requested. The numbers of LEAs varied 
from statement to statement, but general Tzat ions, can be made for 
areas where DOES could provide support to LEAs. The support 
may take the form of sample or model forms for follow up surveys 
of employers and students, providing the OE codes so they are 
readily available for LEAs to determine t^e number of student 
employment options, and assisting LEAs to provide data on selection, 
training, and placement by females and males, so ratios or indexes 
are available to check for sex discrimination. Also, since the 
"attainment of project objectives" vas highly rated by all groups, 
well-stated measurable project objectives should be identified from 
a wide range of projects (with different emphasis, such as instruc- 
tional and equipment) to have some commonality among the objectives 
checked as being attained. The DOES criteria for measurable project 
objectives is that two staff can agree that they are measurable; 
the definition used in the Questionnaires for this project stated 
that a measurable objective was one which resulted in a count, pro-, 
portion, or other' quantitative summary of data. Crnsiderinq sample 
statements provided by LEAs, there is a need to provide modil state- 
ments over a wide range of project objectives. 

In the summary of the results for the predictive impact state- 
ments it was also not JPthat several steps might be helpful in pro- 
viding support to LEAs to implement the .col 1 ection of information 
for the priority statements. The majority of LEAs were able to 
provide some information on local job openings, but this was variable 
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f^-' It might be useful to again. provide examples of the sources of data 

for job openings. and the manner in which the data can be compiled. 
OE codes weKe' sometimes available and sometimes not, and. need to 
be provided to LEAs or- they should be informed of how to obtain 
the. codes. The two indexes for which sample forms and procedures 
should be developed are the employer ratings of entry level skill 
requirements and the surveys of stu4^t interests. Model forms and 
procedures would assist LEAs to collect the information to substan- 
tiate the program need in terms of student interest and up-to-dateness 
with- respect to employer needs. Samrfle statements of measurable 
projec^t/objectives are also needed flbr LEAs. Many of/these support- 
ing documents and procedures can IJe developed by IRDOE i n conj-u^t ion 
with DOES, and then also discussed and examined by representatives 
of LEAs, both BOCES and large cities. This will be espec ial ly ' val - 
uable if the evaluation and funding handbook is developed, as recom- 
mended below. - . 

The results of the analyses of data available and the distri- 
butions of indexes developed for each impact statement were used 
to develop a sample rating form; tentatively titled: Project quality 

.[,. -Criteria: impact . The form is reproduced in this report as Figure 5, 

and contains seven impact statements with scale categories and weights 
for each each of the seven statemisnts . The.wei ghts for the statements 
were derived for the ranks of the statements made by DOES supervisors 
. (see Appendix G). The data required from LEAs are feasible given 

advance informr^tion on the specific pieces of information requir^. 
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la fact, LEAs could a 1 so be given the rating forms with the descrip- 
tion of the indexes and weights to provide the Information ready 
J for summary by DOES. .. 

A brief review of the FY 1975 DOES gating forms and screening 
criteria resulted in several recommendati(Dns giVen In Part V of this 
report. - The main recommendation was to sepaY^ate^the minimum standards 
or screening statements in the rating forms from the implicit quality 
rating and to make explicit statements of the quality statements that 

need to be rated In the areas of Management/Planning and Instruction/ 

Equiprnent. The Next Steps, discussed below, include the|se ' recommendations 
Next Steps 

The next steps needed as a follow up to the ratings and weights, " 
developed for the predictive impact statements are as follows: 

1. Review by DOES of the impact rating form and weights'; 

2. Review by DOES of the recommendations for the screening 
criteria. Management and Planning Qual I ty Rat i ngs , and 
instructional /Equipment ratings; 

\^3. Meeting with IRDOE staff to consider procedures/rating 
\ statements if weights. are desired for the Management/ 
Planni ng. and Instruct ional /Equipment rat i ngs (revi si on 
of statements and rank order form); and 
k. Devel6|iment of the other quality rating forms and 

weights and needed revisions of the Impact rating form. 
After these steps are carried out, it would, be useful to review the 
procedures and data with groups of LEA directors_or representative 
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directors.. This step will .continue the dialogue started with the * ' 
questionnaires, but will also ensure further LEA' input into the 
substantive areas of the project, informing them of the pres- 
sures for the effort to provide data for both funding decisions and 
eval uat ion (outcome) of grants awarded. At this stage it would also 
be useful to include discussions of the Vocational Education Data 
System (VEDS) and be sure that the impact data are included in a 
form amenable for the VEDS system, al so, 

Fqilowing, up on these discussions, a funding and evaluation 
handbook should be prepared for LEAs. This handbook will include 
sample forms and procedures for all the major data needed for the 
impact ratings and the VEDS system. The funding and evaluation 
handBi)ok should be developed in consultation with DOES and LEAs, 
piloted with LEAs to obtain format ive eval uat ion feedback on the 
handboo^k itself, and' probabl y prepared in a loose-leaf format so 
that changes and additions can be made by DOES without requiring 
a new publication. The development of ' such a handbook should 
serve several purposes,- including improving the funding decision 
making process, assisting LEAs to provide' more accurate and useful 
data in. both applying for grants and providing. evaluation of completed 
grants,' and further improve program qtial i ty by helping DOES to 
identify LEAs where special effort should be devoted to assisting 
them to meet VEA grant requirements. 

In summary, the recommended next steps are designed to meet 
the evaluation requirements o/f PL 9^-^82 and, to provide the foundat ion 
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for "rnstruct.ional" materials for LEAs directed toward improving the ■ 
"impact" of VEA-supported. programs and projects. The recommended . 
handbool< and data collection for funding and evaluation will assist 
DOES in supporting projects designed to meet important outcomes Fn 
vocational education and to assist LEAs to identify and improve 
their grant applications and evaluations. . .. 

In order to provide another form of summary arfd indicate the 
final system that would operate to review project proposals, a flow- 
chart of the process has been made and is given in Figure 8. The • 
figure shows the outl ine of the review process as it would function 
with the inclusion of the predictive and outcome Impact statements. 
The details of the review process would be those that are currently 
scheduled in the DOES Procedures -for Reviewing VEA Appl ications 
(Appendix A). In addition to the general project review process, it 
was recommended that there be provision for annual review of' the 
weights and scale categories for both the predictive and outcome 
impact statements. Figure 9 suggests this annual revision process, 
again in outline form. The process includes using weights during one 
fiscal year, collecting data on the impact statements after project 
evaluations have occurred, and using the data for all projects to form, 
distributions on the indexes for the impact statements. These distri- 
butions can be examined 'to determine if the scale categories still 
seem appropriate or need to be "raised" or "lowered" on the basis of 
project attainments. As mentioned above, these processes should assist 
DOES and LEAs to improve programs and funding decisions on the basis of 
both "objective" and "subjective" information from many sources. 



Review Process for VEA Grant Applications Using: Predictive 
• , ■ and Outcome/Impact Statements 



Is 

Project \ Yes 
^Continuation 
? 



Lool< at Outcoii 
Impact *for Past 
Project 




ivakate Proposal . 

Projection: 
,i Screening Criteria 
ii Quality Ratings on:' 
^ Manageinent/Planning 
Instruction/Equipment 

'I ii. Impact 

~~7F ^ 




No 



Reject Project Proposal 



Ifi^ERXC- 




hkt Recommen- 
dations for 
Improvements 
(HRI) 



Yes 



^ Yes-v 


LEA Project 


') 


Funded' 







Evaluate Program 
On Outcome Impact 
Criteria 



LEA Local Support 
or Decision to: 
Terminate Project 



7^: 



No 




Yes 



V 



72 



. Figure 9 

Annual Revision of Impact Statement Weights 
and Scale/ Categories 



Predictive Impact Weights 
and Scale Categories Used 
to Rate Proposed Project 



Project Evaluated Using. 
Establ i shed Outcome Impao^t 
Weights and Scale Categories 



Data Col lected for all 
Projects on , Predictive 
Outcome Impact Scale 
Categories 




and 




heights and. 
.Scale Categorie} 

in Last FY 
StI 1 1 Appropriate^ 
? 



New Weights arid Scale Categories 
for Predictive and Outcome Impact 
Statements .Estab.I isheid 




/ 



81 



\ 

\ 



References 



Dueitt, S.B., riark, D.L. , Harris, Z. , and Ory, J.C. To develop land 
field test criteria. Instruments, and procedures to evaluate | ' 
vocationaV education programs in the Metropolitan Community 
Colleges of Kansas City, Missouri . Final Report , Vol ume 1. 
Kansas City, Mo. : Metropolitan Communi ty Col leges Administrative 
Center, Adgust 1977. 

J., and Hogarth, R.M. Confidence in judgment: 
11 1 us ion of val idity. Psycho 1 o^ I ca 1 Rev i ew , in 

Psychometr i c Methods (2nd' Ed.) New York: , 



, Harris, Z. , and Clark, D.L. Constrijction^f^rilerion v(,eights 
— evaluation of vocational education pro^g-rams- 'Paper presented 
at the annual] meeting of the American EducafTional Research Associatibn, 
.Toronto, March 1978. . [ ' '^-'"^ . A 

Ory, J.C. , Harri s, Z.,. Dueitt, S.B., and Clark, D.L. The development and 
field testing of avocational education evaluation model. Paper 
presented at the annuaT meeting of the American Educational Researjch 
Association, Toronto, March 1978. 





,City, 



Procedures Manual . -Final Report Volume 11, Eval uation Model , Kansas 
Mo.: Metropol i tan Community Col leges Administrative Center, 
August 1977. 

Tittle, C.K. , Spi-lman, H. , and Faggen-Steckler , J. An exploratory study 
of ttje impact of vocational education programs : Implications for 
funding. CASE 21-77-^ New York: . Institute for Research and 
Development in Occupational. Education, Graduate School and Uni- 
versity Center, City U^niversity of New York^ August 1977- 

. /' ■ 



82 



-75- 



/ 



APPENDICES 



8.1 



ERIC 



Appendix A 
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\ 

Procedures for Reviewine VEA ADDlications 



Projects batched by planning region during intake process, distributed 
to review team members. 

Review team develops a schedule for review of projects. 

Review team reviews projects on the six initial screening criteria; 
projects meeting all screening criteria continue on in review 
process; projects not meeting these screening criteria stopped 
until problems, are resolved. / 

Review team determines the primary target population of the project 
(genersd., adult, disadvantaged, handicapped); target population is 
noted onceover page of management program quality scales . 

a. Review t;e,is» rates projects^ for general and adult p^^^ilations 

on the cicpK^ ^ic /manpower needs screening criteria . Disadvantaged 
and hsndSc^app^d projects designed to give students job skills 
- shou.1:i ai«*0 rated on the economic /manpower needs screening 
critRi'ia:; alA projects are^ placed in high, medium and low 
categortdLi* 

b. For projects with disadvantaged and handicapped populations that 
axe designed to give sttidents remedial and preparatory instruction 
and services, the manpoxcer needs criteria should be considered 

as part of the management program quality criteria (see step 9). 

a. For projects in high or medium categories on the economic/manpower 
ne?ris screening criteria ; 

(1) Review team determines program strategy assignment for 
the project using the . following criteria: 

continuation project: those projects previously funded 

under VEA for which the Education 
Department has a moral obligation 
(up to 3 years) 

. improving existing- those projects designed to improve 

programs: / existing occupational programs for 

^ / adult, disadvantaged or handicapped 

populations 

improving existing there projects designed to improve 

programs: existing occupational programs for 

(equipment emphaal.^; geacral nopulations which have 

requet:*:s for equipment that amount 
to 50% or greater of the total 
project ' ost 
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improving existing those projects designed to iiaprove 

programs: existing occupational process for 

(staff emphasis) general populations " which have 

requests for equipment that amount 
to less than 50% of the total 
project cost 

new programs: those projects that propose program 

activities not previously offered ^ 
by the local agency 

(2) Review team determines technical reviewer assignment 
(except for general population projects which are for 
work study) . 

\ (3) Review team determines if a special reviewer (handicapped, 

V adult handicapped, bilingual) is required; plroject assigned 

\ to the special reviewer • 

■ ^\ ' '• . 

b. For projects in low category .on mdnpower needs screening criteria: 

(1) Review process is stopped. 

7.. a. Copy of the proposal. is sent to each technical reviewer assigned 
to the 'project xrf-th requested date for completion of review* 



' b. Copy of the pr6posal is sent to the special reviewer when required 
with requeste^^ date for completion of review, 

3, a. Technical /fevlewer rates project using instructional program 
quality scales * 

. - b,^ Special reviewer comments on project using the special reviewer 
. Comment sheet. t 

\ ■ ■• 

,9. Review team rates project using management proeram quality scales 
(for disadvantaged and handicapped projects that are designed to 
give student renjedial and preparatory instruction and services, 
the manpower needs criteria should be used as part of the 
mana<gement program quality scales ) . 

10« Technical and special reviewers return proposal and review sheet 
to reviewer by completion date. 
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11. Review team meets to discuss project after! all review sheets are 
completed. 

a. Discrepancies In ratings are discussed and resolved. 

Technical reviewer ratings and special reviewer comments 
are discussed and request made to technical and special 
reviewers for consultation when ratings or comments 
appear to be in conflict with other ratings- of project. 

c. If necessary, requests made to educational agency for 
clarification of certain points by review team member 
responsible for that particular agency. ^ / 

d. If conflicts in. ratings exist that cannot be reconciled, 
project review sheets and summary of differences are 
sent to the arbitration panel (D. Post and M. Van Ryn) . 

e. Arbitration panel review information available and makes 
determination on what score the project should receive, 

f . Arbitration panel notifies review team of final racing. 

12. Review team gives project an overall ranking based on/ program quality 
ratings within program strategy assignment (i..e. continuation. 
Improving existing programs, new programs). / . 

13. Review team ranks projects across program strategy assignment within . 
economic/manpower needs category (i.e^'^iigh 1, 2, 3. ..n; medium 1, 2, 3.j.n) 

14. Ranked projects are applied against- dollar . allocations for the region; 
summaries for region are developed. 

*15* . Review teams meet wit;h Bel ton and M= Van P.yn to present suciniaries 
for regioELS. 

*16. 'Budget negotiations with applicants are conducted by the member of 

the review team responsible for that Agency; weak aspects of the 
proposal are revised during this time by the applicant (e.g. items 
receiving low ratings). 

*17. Review team discusses negotiated budgets and proposal revisions; 
final recommendations made. 

*18. Recommended projects with negotiated budgets are submitted to 

D. Belton (secondary) and M. Van Ryn (postsecondary) for review 
and approval (Belton should be carboned for postsecondary projects; 
Van Ryn carboned for secondary projects). 



So 
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*19. 



*20. ? 



Approved jirojects are packaged and grant award letters are prepared 

'^f?''!/-^^"'' availability. If funds are 

avk^able. proposal and grant award letter sent to Assist^t 
avrSfh^r" final app^^^i signature. If funds are not 
bure^t P'^oposal and grant award letter retume'd to appropriate 

*21.. Assistant Commissioner approves project." " 

*^22. Approved project re tume-d to finance officer to be logged in. 

S^Lancl'Jf,-" "^i'"' '° applicant; copies of letter sent ' 
CO finance officer and responsible- bureau chief. 

■ "N^ 




* These steps occur after April 21. 
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Appendix B 

QUESTIONNAIRE 1: TIANKING AND RATING OF IMPACT STATEMENTS, 

Instruct ions for Outcome Impact Statements * — • Rank Order 

.. Listed below are statements dealing with the possible impact of 
VEA projects eligible for financing from the State's "Basic Grant*' 
monies. Please rank these statements in the order of their importance 
as project outcomes. 

. Write a "1" in the right hand column opposite the statement that 
you feel describes the most Important impacts a *'2" opposite the secpad 
most important statement, etc. until all 1.0 statements have been ranked. 
The least important statement of impact of a- project wi 1 1 be ranked "10. 

Outcome Impact Statements ^ Rank Order 

Program graduates are working in occupations for which • 
they were" trained. 

Project objectives are fulfilled. 

No sex discrim-i nation occurred in student 
selection, training, and job placement. 

Training objectives are met In the most cost 
effective manner. 

Large numbers of students are trained. 



Training increases student employment options. ... 
Employers- are sat isfied'with graduates of program. - 
Students trained have positive attitudes toward work. 



Students trained cent inue thei r education. 



Program can be repl icated in other LEAs. 



Please add any outcome statements that you feel are 'Important 
and should be incl uded, ..but do not rank them with the others: 



Institute for. Research and Development 

in Occupational Education 
Center for Advanced Study in Education 
— Graduate School 6 University Center, CUNY 



' * ' B 2 ■ • . 

Instructions for Rating Outcome Impact Statements ^ 
■ • ■' \ ' " 

Listed below is a set of outcome impact statements. How important, 
is each outcome? Give each statement points on a scale from 1 to 20 . 
where . ' . 

20 represents a highly important outcome, the maximum number of points 

1 represents a very unimportant outcome, the lowest number of points 

Write the number of points from 1 (low rating) to -20 (highest rating) 
opposite each of the following impact statements according to its importance 
as a consideration in the funding of VEA projects. 

Outcome Impact Statements Ratings 

1... ...20 



Lowest Highest 



Progr:<5n graduates are wcrrking in occupations 
for which they were trained. 

Project objectives are fulfill led. 

No sex discrimination pccuF^red in student 
selection, training, and job placement. 

•• •■■ ■ \ 
Training objectives are met in the inost 
cost effective manner. 

. ■ \, 

Large numbers of students are traJnedc\ 

Training increases student employment options. 

i . ■ \ 
Employers are satisfied with graduates 
of :^'prDgram. \ 

Students trained have positive attitudes 
toward work. 

Students trained continue their education* 
'Program can be replicated In other LEAs. 



\ 

\ 



\ 



8a. .• - . \-. 
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Instructions for Predict We Impact Statements ^- Rank Order 



Listed below are statements dealing with predictive Impact, that 
is, outcomes: which might be expected to result if proposed VEW projects 
were to be Implemented. P.lease rank each of these statements in, the 
order of their importance as pbss I ble outcomes to be considered In 
making funding decisions. 

I Write ia in the right-hand column opposite the statement you 
fee); describes the most important type of predictive impact., a ''2'* 
opposite the second most important statement, etc., until all 10 state- 
ments have been ranked. The J^ast important statement 'cf predicted 
project Impact wi l I be ranked 'MO**. . 

Predictive Impact Statements Rank. Order 

Students will. be trained for occupations where Jobs 
are ava i lable. / 

/ \ . — 

Project objectives are stated In measurable terms* 

" / ' \ ! — 

No sex discrimination will be made in recruiting 

and placing students In vocational programs. r 

Training objectives will be met in the most cost 

effective manner. . . . - 

Large number of students will be trained. ^ 

Trslning will be--p^ovided to Increase students' 

employment options. . - 

Students will be prepared to meet entry level skill 
requirements as specified by prospective employers, 
(e.g. , employer's ratings of performance objectives 
in terms of job requirements). 

Program will serve students ' -interests. ' 

Program is articulated with local post secondary 
institutions. 

Program will be replicable In other LEAs. 



Please add any predictive impact statements that you feel are 
important and should be Included, but do not rank them wl th the. others . 



instructions for Rating Predictive Impact Statements-. 



B i» 



■Listed below is a set of predictive impact statements. How important 
IS each predictive outcome?. Give each statement points on a scale from 
I to 20 where ■ 

20 represents a h i oh ly "important "^redlrMv^ statement, the maximum 
number of points. i 

1 represents a verv unimportant pr^dlei-.v^ statement, the lowest 
number 'of points. 

Write the number of points from 1 (low rating) to 20 (highest rating) 
opposite each of the following predictive impact statements according to 
Its importance as a consideration in the funding of VEA projects. 



Predictive Impact Statements 



Students will be trained for occupations where jobs 
are aval lable. 

Project objectives are stated in measurable terms. 

No sex discrimination will be made in recruiting ' 
and placing students In vocational programs.. 

Training objectives will be met in the most cost 
effective manner. 

Large number of students will be trained. '• . i 

Training wi'll be provided to increase students' 
employment options. 

Students wl 1 1 be prepared to meet entry level skill 
requirements as specified by prospective employers, 
(e.g., employer's ratings of program performance 
objectives in terms of job requirements). . 

Program will serve students' Interests. 

•Program is articulated with local post secondary 
institutions. . . ^. -.^ 

Program will be repllcable i.n other LEAs. 



Ratings 

1 .20' 

Lowest Highest 



■ 1- 
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QUESTiONNAIRE 2; PREDICTIVE IMPACT STATEMENTS 
Spring 1978 study 

Please complete this page for the project proposal listed In the letter. 



Project Title 



Name of Agency^ 



Project VEA Number 



The present grant covers 



Fiscal year of grant 

% of the program's costs. 
Grade level (s) of program (circle): 9 10. .11 12 postsecondary 
Student type: 

No. ^% Dlsadvan- No, 



adult 



General 



% Disadvan- 
taged* 



% Hand-l- 
capped 



No. 



This project is to bei funded In the fol lowing (purpose) category (s) 

GeneralQJ Disadvantaged* | | Handica 

The grant IS primarily for | [ a continuation project a new program (not 

previously offered 



PPedj I 



Improving existing programs wi th 



I { staff emphasis 
(equipment costs 
less than 50% of 
project costs) 



If a coat in uat Ion project, which year is the project? 



by LEA) 

L j equipment 
emphasis . 
(equj pment 
costs»50% 
or more of 
the total 
project cos 



rYear 



of a 1 or 2 or 3 year project' 
(circle one) - 



Project Summary:' It will be helpful in Interpreting your questionnaire 

pfL'se'attach'^r ' ''""'t^' description (a 2o5-30b word summar^) 

pjease attach the summary to the completed quest lonnaii 



re. 



*Persohs (other than handicapped persons) who have academic or economic 
disadvantages and require special services, assistance or programs in 
order to enable them to succeed in vocational education programs. 

Institute for Research and Development - 

jn Occupational Education . • - . 

Center fo? Advanced Study in Education 
Graduate School S University Center, CUNY . 
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PREDICTIVE; Students will b(. tTa}ned for n r .cupat.ions wh^r^ iob. .r. ....ti.ki, 

^' D^oi!^;? ^'fP^M-'"^' be available for students trained In this, 

project? (Employment data collected between and 



mo. /year 
Number of Jobs 



.) 



mo. /year 

Data Source 

Newspaper ads 

|State employment office 

Private employment agencies 

i Direct contact of employers 

\ (Number of employers contacted ) 

iLEA Placement office 

Unions 

Other (describe source) 



Total number of Jobs 



2. How many Jobs are or will be available In the region? 

State employment office 

— Other (describe source) 



/ 



Total number of regional Jobs. 



PREDICTIVE: Pfolect ah\..tl.. s are st.t^. In measurable t.rr.. 



Number of measurable* project objectives 
Total Number of project objectives 



/ 



*as"a"^counf'oroni:^-^ defined as a project ^dutcL that can be stated 

as a count, proportion, percentage or another quantlt/y. 



ERJC ivii ti i-U ): I u.irA-i 



PREDICTI VE; No sex discrimination will be made in recruitino and placing 
students in vocatibpal ♦ programs . 

Check each activity below that will be carried out prior to or during this project, 

r I Vocational courses are equal ly available to female and male • 
.•——I students upon request. 

□Female and male students will be recruited In approximately 
equal numbers. 

I . I All course Instructional material will be free of sex bias 
— ' and sex role stereotyping. 



□ 
□ 



Program teachers will equaTly represent females and males. 

''Role models'' of the "nontradi tional'* sex for the occupation 
will visit the program. 

All career materials will be free of sex bias and sex role 
stereotyping. 



Men and women students are provided Information about' their 
rights to equal educational and employment opportunities under 
the law. 



X 



□ 
□ 



A program will be cond«i£ted • for parents which will assist them to 
work with their daughttirs and sons to consider all educational 
and employment opportunities. 

Special support services or counseling will be provided to 
females and males who select a nontradit lonal occupational program. 

Equal emphasis will be placed on financial support and 
cooperative educational placements for females and males. 

* . , ! 

instructional and related guidance personnel wi 1 1 be 

□ provj'ded the Inservlce training necessary for the delivery 
of sex-fair instruction and counseling for students. 

PREDiCTivE: Training will be provid ed to increase student's emDloyment options . 

Use the OE code to .1 1st, the number and tittles of . occupational areas for which 
graduates of this project are prepared (4. g., 07. 090i» Medical Assistant, 07.0906 
hea 1 tn Aide^. 



Estimated Project Costs Listed In'VEA Budget 



All of the following questions pertal.n only to the est imated costs of the program 
f inanced by the VEA grant and for the durat;ion "^f the VEA grant. Use the FA-10 
Budget Summary page for the totals below. 



3. 



Current VEA Costs. Current costs are tlie monies from codes 200 (200-250 
'Instruction), 300 (Guidance), ^00 (Healtlh); 500 (Transporta/ion) , 600 
(Operation), 700 (Maintenance), 830 (Renltal), 900 (Food), 1000 (Co-curr ic.) , 
1100 (Comm. -H rela.), and Indirect costsl 
(FA -10 Summary Page) \ 

Total Current Costs.. ...•..••....$ " 



2. Capital VEA Costs . Capital costs are the monies from codes 260 (Staff develop- 
ment) , 1220.3 (Minor remod.), and 1230 (EqVjip,) J 



- Total Capital/^Costs. .•$_ 

Total amount of VEA grant (Grand Total 1+2) . . . , . T."", ....•$ 



'f. Number. of students that are expected to participate 
in the project 

5> . Number of hours of instruct ion that each student is 

to have (hours per week, times number of weeks!) 

(There are the equivalent of 36 instructional weeks 
per school year) 

6. In order td qualify for the job for which the broject 

is preparing the students, will they have to/ continue training 
beyond the time covered by the grant (one year)? 



Yes 



No 



□ 



7. If yes., for how long? 



PREDICTIVE: Students wi 1 1 be prepared to meet entryj-leveT ski 1 1 requirements as 
specified by prospective employers . 

Agreement between number of essential entry level sklills checked by occupational 
advisory committee of employers and union representatives and number of essential 
entry level skills listed by LEA. 

a. Number of skills listed by school 



b. Number of skills checked as essential 
entry level skills by. employers (and/or 
advisory committee) 



c. , Number of employers (committee members) 
reviewing essential skills list. 



i 



v./ ' 



PREDICTIVE; Program will be replleable In other LEAs. 



9^^Q k «ach Item that could be reproduced and sent to another school 




□ 

□ 
□ 



course outl ine 
lesson plans 



project-developed stSedent workbook/student Instructional 
materials /' 

student c>s_sessment forms (tests or rat^hg forms) , 

project-developed slides/AV materla/^f llms 

list of recommended equipment/te«books/AV materials 

list of suppliers and venders/ 

teacher program guide / / 



project proposal 



□ 



I 1, ■ ■ - 

program evaluation report— data and sample forms 



□ 



Total Number checked 



Bas^d on theTtims "checked above, what percenl'^'o f the pro gram do you 
estimate] wl 1 1 be transferable to another school? 



PREDICTIVE; Program will s4rve students' Interests / ; 

Data Source: A Career interest survey can be conducted to provide an 
estimate of the fol lowing numbers. 



_/ ' 



jK Total number of students surveyed 

2 . - -Number-of- students -sel ect I ng th i s 

occupational iarea as first or second- 
choice {of Interest to them). 



. : PREDICTIVE:. Program Is articulated with lociial post/secondary . institutions 

Number of local post secondary instltutiorfs Where 
j- students could continue in the same occupational 

■ . • field, • . . . / • 



